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ABSTRACT 

The 1975 evaluation report for the Career Education 
Instructional System's exemplary project at Newark School District, 
Delaware, describes the evaluator's activities in accomplishing four 
tasks. In the first task, each of the 155 curriculum units developed 
by the project was classified according to instructional goals by 
school, grade level, and title and related to the federal guidelines 
for evaluation of career education projects. The second task 
identified subob jectives appropriate to student groups. Preparation 
and implementation of the process, product, and side-effects 
evaluation activities of the evaluation plan are described in the 
third and fourth tasks, including document review, interviews with 
project management and teachers, and random group testing. In 
addition, evaluation of the three project goals — community 
involvement, curriculum development and implementation, and 
dissemination of project materials— is presented. Recommendations for 
project improvement included continued development of individual 
career guidance components, continued expansion of curriculum 
materials, and increased emphasis on evaluating product outcomes. 
Most of the document consists of appended tabular evaluation data. 
(MF) 
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INTRODUCTION 

The information in this secona annual evaluation report is based upon Educa- 
tional Testing Service's involvement with the Newark, Delaware Career Educa- 
tion Instructional System's (CEIS) Project from January 1, 1975 through 
December 31, 1975. The purpose of the evaluation is threefold: first, to 
assess the extent to which the goals of the project have been achieved; second, 
to assess the extent to which project-related activities have produced side 
effects; and third, to provide feedback to the project management for decision- 
making. 

In planning and conducting the evaluation, an actempt was made, as closely 
as possible, to conform to the Federal Guidelines for evaluation of career educa- 
tion projects. During this period, the evaluation was under the direction of 
Mr. Raymond G. Wasdyke, a project director in ETS's Elementary and Secondary School 
Programs Division, and Mr. J. Robert Cleary, Associate Director of ETS's Consul- 
ting, Advisory and Field Services Division. 

CEIS PROJECT REVIEW 

The CEIS project is a federally sponsored project which is entering its third 
year of operation. During its second year of operation, it involved the parti- 
cipation of approximately 800 students in grades K-3, 1,500 students in grades 
4-6, l,n00 students in grades 7-8 and 500 students in grades 9-12. The project 
is basically developmental in nature with implementation of the curriculum and 
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other materials developed during the life of the project occurring according to 
a schedule determined by the teachers. 

The project's goals focus on the involvement of a broad spectrum of the 
community in planning, developing, and implementing the project, developing and 
implementing a kindergarten-through-eigth-grade career education curriculum, 
and disseminating the materials developed by the project. 

EVALUATION ACTIVITIES 

ETS's evaluation activities during the second year included the accomplish- 
ment of 4 specific tasks as outlined in its proposal. Some of the tasks were 
directed at the development of specific v;ork products, as required in the 
Federal Guidelines , while others were more process-oriented. The following de- 
scribes ETS's activities related zo these' tasks. 

Task 1 > Completion of Treatment Group Outcome Area Tables 
The Treatment Group Outcome Area Tables (TG/OA) plot and classify each of 
the 155 curriculum units developed by the CEIS project. Each of the units de- 
scribes the implementation of career education concepts- in a unique way. Also the 
decision to implement each unit is determined by the teacher. Therefore each 
unit can be viewed as a separate treatment. In essence, then, the TG/OA tables 
contain 155 treatments 

The TG/OA tables contain the following information for each of the schools 
involved in the CEIS project: 

(1) Name of teacher implementing the unit 

(2) Unit's grade level 

(3) Unit's title 

(4) Instructional goals covered by each unit 
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id i;i Appendix B of the evaluation plan, pages 34 through 52 J Since iii/UA 
les are included as part of the evaluation plan and to avoid redundancy, 
/ are not presented in this report as a separate appendix. 
Task 2. Completion of Outcome Question Trea^-'^ent Group Matrix 
The Outcome Question Treatment Group Matrix (OQ/TGM) identifies the speci- 
sub-objectives of the Guidelines' design which are appropriate to student 
jps. Teachers involved in the CEIS project provided the information in the 
"GM tables. The OQ/TGM tables appear in Appendix C of the evaluation plan, 
;s 53 through 99. 

The completion of the TG/OA and OQ/TGM tables demonstrates one of the funda- 
ial weaknesses of the Guidelines . Both of these products focus on the antici- 
5d effects of treatment to student groups and exclude non-student groups. Con- 
iently, if one restricts evaluation activities to those required in the 
ielines ; the specification of intended outcomes for non-student groups would 
be included. However, since the CEIS project deals with non-student outcomes, 
n'sions were made to include these in subsequent evaluation activities. 

Task 3 . Preparation of Evaluation Plan 

ETS developed an evaluation plan that conformed to the Federal Guidelines 
evaluation where reasonable and yet was responsive to the unique characteris- 
; of the CEIS project. The plan's preparation included a thorough review 
ihe Guidelines and CEIS project documents. Interviews with CEIS management 
Ff and teachers were also conducted to provide yet other sources of information. 
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ETS views the plan as a developmental one, increasing in precision and sophis- 
tication over the tenure of the project. Due to the length of the evaluation 
plan, it is not included in the main body of the report, but is found in 
Appendix A. 

In brief, the evaluation plan serves the purposes of the evaluation, as 
mentioned earlier. (See Introduction p. 1) Process, product, and side-effects 
evaluation .activities ar^ outlined in the plan, as well as data collection 
sources from student and non-student groups. Process evaluation is designed to 
provide information to the project management staff for decision-making purposes. 
Product evaluation focuses on student assessment and is embedded in a pretest- 
postest design using treatment and non-equivalent comparison groups. Side- 
effects evaluation is intended to assess the worth of unanticipated project 
outcomes. Selection of student and non-student groups and events to be inves- 
tigated were determined by random-selection procedures and in close cooperation 
with the CEIS project staff. 

Task 4. Implementation of Evaluation Plan 

The intention of Task 4 was the implementation of the process, product and 
side-effects evaluation activities specified in the evaluation plan. Because 
the CEIS project is funded on a calender- rather than on a fiscal -year basis, it 
^as impossible to meet all of the requirements for evaluation specified in the 
plan. For example, since many of the treatment groups were exposed to some sort 
Df project-related activity prior to the approval of the plan, pretest and postest 
iata were not collected on a large scale. 

ETS's findings,, conclusions, and recommendations are grouped under each 
Df the CEIS project's broad goals as stated in its original proposal. 
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CEIS GOALS 

Goal A 

Given the opportunity, time, and funding necessary for continued involvement 
of the total community (business, industry, and education, parents and students) 
in the on-going development of the Career Education Instructional System, com- 
munity participants will be able to: 

!• Provide advisory services through participation as members of the Career 
Education Project Advisory Council. 

2. Directly participate in program development by serving as members of 
elementary, midtfle, and high school Career Education Ad Hoc/Task Force 
Committees. 

3. Serve as resource consultants to career education program development 
teams and career-related classroom activities. 

Findings and Conclusions 

ETS attended two Career Education Project Advisory Council meetings, inter- 
viewed select Council members, and reviewed Council and Ad Hoc/Task Forces docu- 
ments as a means of gathering data pertinent to the stated goal, ZJS also per- 
formed a discrepancy and verification analysis between planned and completed 
management activities associated with this goal area. 

The findings of this analysis indicate that the functioning of the Council 
has been guided by the project's management plan. Meetings have been held on 
schedule. Ad Hoc/Task Forces have been established and are operating as intended, 
and the planned Council reorganization has taken place. During the past year, 
the Council has shifted from its initial efforts to get the project off to a 
jood start and has taken a policy making and task-oriented approach. 

The Council's agenda for 1975 included the following: 

- Identifying priority areas for career education in the Newark district 
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- Monitoring the operation of the CEIS project 

- Initiating policy about the scope and expansion of career education in the 
district 

- Serving as communications interface between the schools and community 

- Facilitating and focusing community input relating to career education 
To accomplish these and other activities, the Council has established the 

following task forces: 

(1) Community Task Force 

(2) Dissemination Task Force 

(3) Guidance Task Force 

(4) Curriculum Task Force 

The most significant achievement of the Career Education Project Advisory 
Council was the development and implementation of the Educational Resources 
Association (ERA) as a cooperative effort between the Council, the Newark School 
District, and the DuPont Company. ERA was formed in response to teachers' re- 
quests for up-to-date and accurate information about the availability of resource 
assistance for implementing career education programs. ERA provides a link be- 
tween the community and the schools thorough the identification and cataloging 
)f more than 1,000 sources of assistance that are available to teachers. In 
iddition to providing the cataloging, ERA acts as a liaison between the teachers 
ind resources through a centralized agency. The findings of 'a thorough inves- 
tigation of Council and project documents, interviews with select Council mem- 
)ers, and ERA staff indicates that ERA was initiated, developed, and implemented 
>y the Council. Therefore it can be classified as a positive, unintended outcome 
»f the project. 

In conclusion, the project is to be commended for carrying out activities 
esigned to meet the project's goal concerning involving the community in the 
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on-going development of the CEIS project* The formation and implementation of 
ERA deserves special mention. 
Recommendations 

The continuing growth and development of the Career Education Advisory 
Council is attributable to the leadership of the CEIS project direction staff. 
During the upcoming year when %deral support for the project will end and de- 
cisions will have to be made abo^t continuing the project, their leadership 
will be tested even more. Although it is unlikely that the project will cease to 
exist after the federal funding has been discontinued, the role of the Council, 
project staff, and ERA will certainly be changed. To ensure that the transition 
from project to a more formal status will occur smoothly, it is recommended 
that the Council and project staff consider as a key item on their 1976 agenda 
the issues, problems, and decisions that will be brought to bear on the Council 
when federal funding is terminated. 
G oal B. 

Given the opportunity, time, and funding necessary for continued development 
and implementation of a kindergarten-through-grade-12 Career Education Instruc- 
tional system, project participants will be able to: 

1. Develop guidelines for career-related curriculum development at the ele- 
mentary, middle, and high school levels. 

2. Develop career-related programs to be implemented at the elementary, 
middle, and high school levels. These programs will include (a) perfor- 
mance objectives, (b) student activities, and (c) academic-vocational 
interdisciplinary approaches. 

Findings 

ETS's findings are based on the results of a teachers' survey and on-site 
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visits to the following schools: Brookside Elementary School, Leasure Lower 
School, Maclary Elementary School, McVey Elementary School, Central, Gauger, and 
Ogletown Middle Schools, and Christiana, Glasgow, and Newark HighSchools, Select 
members' of the instructional and administrative staff were interviewed in each 
of the schools visited. 

The teachers' survey was conducted in the spring of 1975 for the purpose 
of providing the project management '^t.-i^f with process and formative data. The 
information gathered in this survey included: sources of information provided by 
the CEIS project staff; the amount of feedback requested from the project staff 
concerning the development and implementation of the curriculum units and the 
degree and type of modification needed to improve the overall quality of the 
units; opinions of how well students liked the units; sources of assistance 
for improved curriculum implementation, project constraints, instructional 
methods used in presenting the curriculum units, and the emphasis of the curric- 
ulum units. 

The findings based upon returns for 91 percent of the 57 teachers involved 
in the CEIS project, are summarized below: 

(1) The most frequently reported sources of information provided to the 
teachers by the project staff were project reports, textbooks, instruc- 
tional manuals and other printed reference materials. 

(2) The teachers reported that the project staff requested feedback from them 
about three times per school year. 

(3) The teachers reported that the curriculum units developed did not re- 
quire any significant degree of modification before being disseminated. 

(4) Seventy percent of the teachers reported that the students liked the 
units very much, and 30 percent reported that the students liked the 
units somewhat. 
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(5) The three most frequently reported sources of assistance requested by 
the teachers were funds for purchasing special materials, release time 
for instructional preparation, and inservice preparation. 

(6) The most frequently reported types of constraints were lack of suffi- 
cient funds and lack of inservice preparation. 

(7) The teachers reported that over half of the curriculum units developed 
are related to improving student growth in each of the following areas: 
self-awareness; basic academic and vocational skills; work values; know- 
ledge of the world of work, and career decision making. 

A complete discussion of the survey's methodology, findings, and results 
are found in ETS's second Quarterly Report, Appendix B. 

Consistent with this year's emphasis on dissemination by the project staff, 
the evidence suggests that the curriculum units have been installed to a signi- 
ficant degree in participating schools. The project direction staff have deter- 
mined to influence career education at the secondary level through the develop- 
ment and implementation of a career guidance system. For this reason, curriculum 
units comparable to those found at the elementary and middle school levels are 
not present at the secondary level. However, an extensive and varied amount of 
guidance materials, primarily for student use, are present and easily available 
at the secondary level. Evidence as defined here consisted of the presence of 
bulletin boards and corridor displays focusing on career education themes, stu- 
dent and teacher career education resource centers, teachers' logs of instructional 
activities related to career education, teachers' utilization logs of ERA, audio- 
visual materials related to job information, and other materials related to 
career education. 

A review of the project's management plan indicates that the project staff 
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were responsive to teachers' requests for increased inservice and workshop 
activities. Summer workshop activities were designed to provide teachers and 
counselors with the opportunity to modify and refine the curriculum units that 
had been developed previously and initiate work on preparing a Guidance Develop- 
ment Plan. Summer workshops included the participation of 67 teachers and coun- 
selors representing 6 elementary and 4 middle schools, 3 high schools, and 1 
parochial school. Since last summer, the project' staff have prepared trans- 
portable inservice guides that can be used by individuals in leadership positions 
to conduct inservice workshops in non-project sponsored schools. 

A major goal for the project staff this year was the preparation of a com- 
prehensive career guidance plan and the development of select guidance components 
Through inservice workshops and contractual agreements with counselors, a career 
guidance plan was developed and materials describing each of the plan's 36 indi- 
vidual components were prepared. ETS's findings concerning the development and 
implementation activities in support of this goal were obtained by a careful anal 
ysis of the career guidance plan and its components. 

ETS also spent two days on-site visiting career resource centers and inter- 
viewing 26 counselors who participated in the development of the plan or indivi- 
dual guidance components. The findings show that the guidance plan is consis- 
tent with comparable plans and the known attributes of successful guidance pro- 
grams. Analysis of each of the guidance components indicates that the majority 
of them are related to career guidance and that the successful implementation 
of these components in the Newark schools will, in effect, constitute a career 
guidance system. However, these components, as well as the career guidance plan, 
are still in the developmental stage and further refinement and modification will 
be needed to maximize the potential contribution these materials can make to 
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improving the career guidance offerings. 

Interviews with teachers indicated that they are receiving and using the 
program materials and that the counselors had a positive attitude about the 
CEIS project and that they feel attention must be paid to the critical area of 
career development. All of the counselors felt certain that their contribution 
in the development of the guidance materials would be effective in bringing 
about student growth in career development. In no instance did the counselors 
have a less than favorable attitude about their involvement in the project. The 
findings do suggest, however, that the counselors are uncertain about how the 
individual guidance components relate to one another. This is to be expected 
however since the components are relatively new on the scene. 

To prepare for the final evaluation phase, ETS conducted a pilot test of 
student instruments in the late fall of 1975 to determine the extent to which 
the instruments were sensitive to detecting changes in students* behavior after 
exposure to the CEIS curriculum units. A pretest-postest design with intact 
treatment and non-equivalent comparison groups was used. Because the postest 
data have not been collected from all of the classes involved in the pilot testing, 
the findings reported here are limited to one treatment and comparison class at 
the third grade level. Currently, the data are being collected from other third 
jrade classes with data collection at the middle and secondary levels scheduled 
For the spring of 1976. 

ETS, in close cooperation with the CEIS project direction staff, selected 
the instruments to be pilot tested. Figure 1 shows the instruments tentatively 
selected for measuring what students learned as a result of instruction. The 
initial choice of these instruments was based upon careful review and analysis 
)f instruments recoimended in the Guidelines and contained in ETS's test 
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Figure 1 

Recommended Assessment Instruments for 
Career Education Instructional Systems Project 
Newark, Delaware 



Student 
Outcome 
Grade Statement* 



I. Self Aware- 
ness 



IK Basic Aca- 
demic Skills 

IV. Awareness and 
Knowledge of 
World of Work 



Assessment 
Instrument 



Number c 
Question^ 



Type of 
Questions 



Piers-Harris 80 
Children's 
Self Concept 
Scale: "The Way 
I Feel About 
Myself" 

Iowa Test of 
Basic Skills 

Career Educa- 40 
tion Cognitive 
Questionnaire 
CECQ, 1-3 



Yes-No 



Matching 



Test 

Administration 
Time 



15-20 minutes 



Part of Newark 
districtwide 
testing program 

30-45 minutes 



Total Test Administration Time 45-65 minutes 



I. Self Aware- Piers-Harris 80 Yes-No 15-20 minutes 

ness Children's 
Self Concept 
Scale: "The Way 
I Fc^l About 
Myself" 

II. Basic Aca- Iowa Test of Part of current 

demic Skills Basic Skills testing program 

IV. Awareness and Career Educa- 50 Matching, 30-45 minutes 

Knowledge of tion Cognitive Multiple- 
World of Work Questionnaire Choice 
CECQ, 4-6 



Total Test Administration Time 45-65 minutes 



' Statements appear in Guidelines for Evaluation , Developmental Associates, Inc. 
1974. 
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Figure 1 Cont, 



Student 
Outcome 
Grade Statement* 



Assessment 
Instrument 



Number of 
Questions 



Type of 
Questions 



Test 

Administration 
Time 



8 



I. Self Aware- 
ness 



II. Basic Aca- 
demic Skills 

IV. Awareness and 
Knowledge of 
World of Work 



.Career 
Decision 
Making** 



Piers-Harris 
Children's 
Self Concept 
Scale:"The Way 
I Feel About 
Myself" 

Iowa Test of 
Basic Skills 

Assessment of 
Career Develop- 
ment, Houghton- 
Mifflin, (ACD) 
Composed of 
following sub- 
scales: 

1 .Occupational 
Characteris- 
tics 

2. Occupational 
Prerequisites 

5. Career Plan- 
ning Know- 
ledge 

Assessment of 
Career Develop- 
ment, Houghton- 
Mifflin (ACD) 

5. Career Plan- 
ning Know- 
ledge 

4. Career Plan- 
ning Activi- 
ties 

3. Career Plans 

6. Exploratory 
Job Experien- 
ces 



80 



40 

14 

40 



na 



38 



4 

90 



Yes-No 



15-20 minutes 



Multiple 
Choice 



Multiple 

C;o 'ce 



na 



Checklist, 
Multiple 
Choice 
Fill in 
Checklist 



Part of current 
testing program 



15 minutes 

15 minutes 
20 minutes 



na 



20 minutes 



10 minutes 
20 minutes 



* Statements appear in Guidelines for Evaluation, Developmental Associates, Inc. 
1974 



** Tests will be distributed to students on a 
will take units 1, 2, and 5 of the ACD and 
4, 3 and 6. Total administration time for 
minutes, 
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random basis. Half of each class 
the other half will take units 
grade 8 is approximately 65-70 



ERIC 



-14- 



collection files. Final determination of the snstruments to be used in the 
third and final year of the project will be made upon evaluation of the empiri- 
cal data collected from the pilot testing. A brief description of the instru- 
ments reviewed by ETS is found in Appendix C. 

The findings reported here are based upon one intact grade 3 class exposed 
to a CEIS curriculum unit over a six-week period and one intact comparison class 
that did not receive any formalized instruction related to career education 
during the same time span. As shown in Table 1, analysis of the differences in 
the mean gain scores within treatment and comparison groups for the Career Edu- 
cation Cognitive Questionnaire for Grades 1 through 3 indicates that the differ- 
ences in the mean gain scores is statistically significant at the (p<.Ql level). 



Table 1 

Differences in Mean Gain Scores 
withiii Treatment and Comparison Groups 
for CEQ 1-3 



Mean Mean Mean 

Pretest Scores Postest Scores Gain Scores t-Value 

Tn = 21 28.524 31.048 2.524 3.164* 

Cn = 23 28.348 29.044 .696 .64 



Analysis of differences in mean gain and postest scores on the CEQ 1-3 be- 
tween treatments and comparison groups, however, is not statistically significant 



* t significant at well beyond .01 level 
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as is illustrated in Table 2 and Table 3. 



Table 2 

Differences in Mean Gain Scores 
between Treatment and Comparison Groups 
for CEQ 1-3 



In = 21 
Cn = 23 



Mean 
Pretest Scores 

28.524 

28.348 



Mean 
Postest Scores 

31 .048 

29.044 



Mean 
Gain Scores 

2.524 

.696 



t-Value 



1 .358 



Table 3 



Differences in Mean Postest Scores 
between Treatment and Comparison Groups 
for CEQ 1-3 



lean Postest Score Mean Postest Score 

freatment n=21 Comparison n=23 t -Value 

31.048 29.044 1.367 



An analysis was also performed to determine the relationship between member- 
ship in the treatment or comparison groups and mean postest scores on the CEQ 1-3. 
fable 4 shows the correlation between membership and score to be positive but 
lot statistically significant. However, if the strength and direction of the 
:orrelation holds with the larger number of cases in the final evaluation group, 
bhe relationships between mean pretest scores and exposure to treatment will be 
jignificafit. 
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Table 4 . 

Relationship of Mean Postest Score 
to Membership in Treatment or Comparison Group 
for CEQ 1-3 



Grade Correlation 
44 3 .202 

The Piers-Harris Children's Self Concept Scale: "The Way I Feel About 
Myself", was also administered on a pretest-postest basis to the groups. As 
shown in Table 5, differences in the mean gain scores within treatment and com- 
parison groups for the Piers-Harris was not statistically significant. The 
^ean gain scores and the t-value is greater, however, for the treatmer^t than 
the comparison group. 



Table 5 

Differences in Mean Gain Scores 
within Treatment and Comparison Groups 
for Piers-Harris 



Mean Mean Mean 

Pretest Scores Postest Scores Gain Scores t-Valu e 

Tn = 22 56.046 58.364 2.318 1.167 

Cn = 24 57.667 58.833 1 .167 . .504 



Analysis of differences between mean gain scores and mean postest scores on 
the Piers-Harris for treatment and comparison groups indicated no significant 
differences as displayed in Table 6 and Table 7. Although the treatment group 
had a higher mean gain score than the comparison group the difference between 
both groups' mean postest scores were extremely small . 
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Table 6 

Differences in Mean Gain Scores 
between Treatment and Comparison Groups 
for Piers-Harris 



Mean 
Pretest Scores 

n = 24 56.046 

n = 24 57.667 



Mean 
Postest Scores 

58.364 

58.833 

Table 7 



Mean 
Gain Scores 

2.318 

1.167 



Differences in Mean Postest Scores 
between Treatment and Comparison Groups 
for Piers-Harris 



t-Value 



.383 



ean Postest Score 
reatment n=22 

58.364 



Mean Postest Score 
Comparison n=24 

58.833 



t-Value 
• 122 



There is no relationship between membership in either the treatment or con- 
rol group and mean postest scores on the Piers-Harris as found in Table 8. 



Table 8 

Relationship of Mean Postest Score 
to Membership in Treatment or Comparison Group 
for Piers-Harris 



46 



Grade 



Correlation 



,02 



Test-Retest reliability was computed for the CEQ 1-3 and the Piers-Harris 
Dr both the treatment and comparison groups. Table 9 shows the estimates of 
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test-retest reliability. Since test and retest were administered over a six- 
week period, the reliability coefficients are lower-bound estimates and indi- 
cate the tests are highly reliable. 

Table 9 

Test/Retest Reliability 
for CEQ 1-3 and Piers-Harris 
for Treatment and Comparison 
Groups 

CEQ 1-3 Piers-Harris 
Treatment Group -592 -756 

Comparison Group .494 .604 

The findings and results of this limited pilot testing study does suggest 
that the CEQ 1-3 is a highly reliable instrument, is sensitive to measuring 
students' awareness and knowledge of the world of work, and that scores on the 
instrument are related to exposure to treatment. Although the mean gain score 
on the Piers-Harris was greater for the treatment than the comparison group, the 
lack of any difference between the mean pjstest scores for the groups and the 
lack of correlation between test scores and exposure to treatment indicates 
that the Piers-Harris may not be appropriate for use in the evaluation. However, 
final determination of the inclusion of either instrument in the final evalua- 
tion must be postponed until additional empirical data have been collected and 
analyzed. 

Recommendations 

(1) Attention should be directed at specifying objectives of the individual 
career guidance components in terms of anticipated student outcomes. 
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(2) In order to increase the potential of the individual guidance components, 
attention should be given to identifying and then exploring ways that 
these components can be linked together in a logical and systematic way. 
As a spin-off of this process, unmet guidance needs would also be iden- 
tified which could serve as the focus of future guidance developmental 
efforts. 

(3) The identification and linking of existing projects will undoubtedly;Ye^' 
quire the development of a more comprehensive career guidance plan than 
is currently in existence. Thus, consideration should be given to the 
development of a comprehensive plan which contains a description of 
anticipated student outcomes in career development and a planned sequence 
of guidance experiences designed to achieve these outcomes. 

(4) Considerable thought and discussion should be given to the issue of 
career development program implementation in the Newark School District. 
The basic issue is whether the CEIS project's program activities in 
career development are viewed as an add-on or as a redirection of the 
current guidance program. We consider the resolution of this issue as 
fundamental to the successful implementation of the guidance component 
of the CEIS project. 

(5) Attention should be directed to the identification and development of 
evaluation criteria and strategies for assessing the outcome of the 
career guidance components. These criteria and strategies should be 
established in corroboration with the third-party evaluator. 

Goal C 

Given the opportunity, time, and funding necessary to maximize the distributior 
of Career Education Instructional System information, project coordinators and 
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participants will be able to: 

1. Prepare sufficient copies of materials for dissemination. 

2. Disseminate information through the Newark School District and the 
State Department of Public Instruction.. 

3. Provide technical assistance to new project participants. 
Findings 

The results of ETS's discrepancy analysis indicated that curriculum and 
other instructional materials developed by the CEIS project were disseminated 
as specified in the management plan. Project materials were disseminated 
through the American Institute for Research, the ERIC system, the USOE Career 
Library, school leaders, and key district personnel. The materials have also 
been disseminated to the Seaford School District and other select school dis- 
tricts in Delaware. Moreover, there is evidence that the materials are readily 
available and in widespread use within the Newark School District. Also, the 
project has been extensively publicized through print and broadcast media. 

It-was determined, through interviews with teachers and the results of the 
teachers' survey, that the CEIS project staff have provided adequate technical 
assistance to project participants. Assistance is typically provided to teachers 
through inservice workshops, school and district-level conferences, team leaders, 
and on an individual request basis. Periodically scheduled guidance meetings 
provided a way of giving technical assistance to guidance staff working on the 
project. The findings suggest that the materials developed by the CEIS project 
have been disseminated according to the management plan and that an adequate 
level of technical assistance is being offered to project participants within 
the Newark School District. A discussion of the results of the teachers' survey 
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is presented in Appendix C. 

Recommendations : The preparation of the guidance materials are in the same for- 
mative stage of development as the curriculum materials were a year ago. It is 
therefore recommended that after the guidance materials have been thoroughly 
field-tested, steps should be taken to disseminate these materials through the 
same avenues used for dissemination of curriculum materials. 

OVERALL CONCLUSIONS 

The CEIS project direction staff are to be commended for the work that has 
been accomplished during the second year of the project. Significant achieve- 
nents include the reorganization of the Career Education Project Advisory Council, 
the development and implementation of a comprehensive career guidance plan and 
career guidance components, and the widespread dissemination of curriculum 
and other instructional materials. Inservice workshops for teachers and coun- 
selors have also been expanded and a transportable inservice instructional module 
has been developed. 

Recommendations for project improvement include continued development of 
the individual career guidance components, continued expansion of the project 
curriculum materials into additional classes, and increased emphasis on evalua- 
ting the product outcomes of the project. 
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AN EVALUATION PLAN FOR THE CAREER EDUCATION 
INSTRUCTIONAL SYSTEM PROJECT 
NEWARK, DELAWARE 

Introduction 

This paper presents a plan for the evaluation of the Career Education 
Instructional System Project (CEIS) operating In the Newark, Delaware school 
district. The purpose of the evaluation Is threefold: first, to assess the 
extent to which the goals of the project have been achieved; isecondly, to 
assess the extent to which project related activities have produced side ef- 
fects; and lastly, to provide feedback to the project management for decision- 
making. In developing the evaluation plan, an attempt was made, as much as 
possible, to conform to the federal guidelines for evaluation of career educa- 
tlon projects. 

In preparation for developing the evaluation plan, ETS thoroughly reviewed 
the federal guidelines and CEIS project related documents. Discussions with 
CEIS project staff and teachers were also conducted to provide yet another 
source of Information. Specific documents required In the federal guidelines 
to be Included In the evaluation plan were also developed. Like most evalua- 
tion plans, the plan will be modified and Improved during the course of the 
evaluation and should not be thought of as being a rigid and Inflexible docu- 
ment. 

CEIS Project Review 

The CEIS project Is a federally funded project which Is entering Its sec- 
ond year of project operation. During the year 1975, It will Involve the parti- 
cipation of approximately 800 students In grades K~3, 1500 students In grades 
4-6, 1000 students In grades 7-8. The project Is basically developmental In 
nature with Implementation of the materials developed during the course of the 
project occurring on a teacher self-selection basis, 

CEIS project goals Include: 
Goal A. Given the opportunity, time and funding necessary for continued In- 
volvement of the total community (business. Industry, and education. 



EKLC 



27 



parents and students) in the on-going development of the Career 
Education Instructional System, community participants will be able 
to: 

1. Provide advisory services through participation as members of 
the Career Education Project Advisory Council. 

2. Directly participate In program development by serving as members 
of elementary, middle, and high school Career Education Ad Hoc/ 
Task Force Committees. 

3. Serve as resource consultants to career education program devel- 
opment teams and career related classroom activities. 

Goal B. Given the opportunity, time and funding necessary for continued de- 
velopment and Implementation of a kindergarten through grade twelve 
Career Education Instructional system, project participants will be 
able to: 

1. Develop guidelines for career related curriculum development at 
the elementary, middle, and high school levels. 

2. Develop career related programs to be Implemented at the elemen- 
tary, middle, and high school levels. The^e programs will Include 
(a) performance objectives, (b) student activities, and (c) acade- 
mic- vocational Interdisciplinary approaches. 

Goal C. Given the opportunity, time and funding necessary to maximize distrib- 
ution of Career Education Instructional System Information, project 
coordinators and participants will be able to: 

1. Prepare sufficient copies of prepared materials for dissemination. 

2. Disseminate prepared Informatlor through the Newark School Dis- 
trict and the State Department of Public Instruction. 

3. Provide technical assistance to new project participants. 

In reference to Goal A, the CEIS project has formed a Career Education 
Project Advisory Council with representatives from business. Industry, local 
government, community agencies and education. Meetings have been held and 
the Council has played a major role In shaping the policies and direction of 
the CEIS project. Ad Hoc Task Forces have also been formed and shared a ma- 
jor role In formulating the Instructional goals for the project In the areas 
of (1) career awareness, (2) career exploration, and (3) career exploration 
and specialization. 
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Activities dealing with Goal B have focused on the development of curri- 
culum units in career education. Inservice workshops in the summers of 1974 
and 1975 have resulted in the development of 195 curriculum units. The break- 
down of the number of units developed in each of the project's instructional 
areas is: (1) career awareness (grades K-5) 95 units, (2) career exploration 
(grades 6-8) 60 units and (3) career exploration and specialization (grades 
9-12) 40 units. The units developed for the career exploration and special- 
ization in grades 9 through 12 however, were not viewed by the teachers and 
project staff as being successful and, therefore, will not be used in the 
school yea- 19 75-76. The instructional objectives in the units are well 
stated and consistent with the broad instructional goals of the project, as 
well as the instructional methodology and evaluation sections of the units. 

The curriculum units and other career education materials developed by 
the CEIS project have been disseminated within the Newark School District in 
partial fulfillment of Goal C. Curriculum materials can be found in the pro- 
ject's participating schools career education resource centers and in many 
of the individual classrooms. At this. point however, the units, except in 
some isolated instances, have not been disseminated outside of the Newark 
School District. 

Relationships Between CEIS Instructional Goals and Student Outcome Statements 

The CEIS project conceptualized career education by three phases; an aware- 
ness phase, an exploration phase and an exploration and specialization phase, 
la each of these phases, instructional goals have been developed. In the left 
hand column of Figure 1. are the instructional goals of the project in each of 
the phases. Across the top of Figure 1. is a list of the student outcome state- 
ments in career education which appear in the federal guidelines. 

The matrix in Figure 1. was constructed in order to determine the relationship 
between the CEIS instructional goals and the student outcome statements in the 
federal guidelines. An "X'* in a column under the heading "Student Outcome State- 
ments" indicates a relationship between an instructional goal and a student 
outcome statement. As graphically shown in Figure 1. each one of the CEIS 
project instructional goals is related to one or more of the student outcome 
statements. However, as was stated previously, CEIS project related activities 
are, for the most part, not taking place at the secondary level. The project 
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COMPARISON BETWEEN CEIS INSTRUCTIONAL GOALS 
AND STUDENT OUTCOME STATEMENTS 
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intends to work in this area however, through the development of a compre- 
hensive career education guidance system at the secondary level. Thus, the 
only instructional goal which is currently receiving attention by the CEIS 
project at the secondary level is goal number 10 (i.e. provide guidance and 
counseling to assist in selecting occupational speciality with options). 

The other instructional goals in the exploration and specialization phase 
(i.e. instructional goals number 8, 9, 11, 12), although originally planned 
to receive attention by the CEIS project, are already being met to a large 
degree through the present instructional system. Therefore, to avoid dupli- 
cation of effort the CEIS project direction staff decided to focus the pro- 
ject's attention on the development of a comprehensive career education gui- 
dance system rather than on other existing instructional areas. 

As part of ETS's formative and process evaluation activities conducted in 
the spring of 1975, ETS asked the teachers involved in the CEIS curriculum 
developmental effort to judge the relationship between the units they de- 
veloped and the student outcome statements listed in the federal guidelines. 
The purpose of this task was to determine the major emphasis of the curri- 
culum units as judged by the teachers. ETS also performed this same task. 
A comparison between teachers and ETS judgements yielded a high degree of con- 
sistency. 

As shown in Table 1, 84% (N^ISO units) of the 155 units developed re- 
late to improving basic academic and vocational skills with approximately the 
same percentage found at the elementary and middle school levels. Awareness 
and knowledge of the world of work is covered by 79% (N=120 units) of the 
units. Seventy-one percent of the units at the elementary school level deal 
with awareness and knowledge of the world of work and 86% at the middle school 
level. Seventy-eight percent (N«120 units) of the units deal with awareness 
of work values and the desire to work. At the elementary school level 73% of 
of the units deal with this student outcome statement as compared to 83% at 
the middle school level. Competency in career decision making is covered by 
77% (N» 118 units) of the units with 72% of the elementary units and 81% 
of the middle school units covering this area. Increased self awareness is 
found in 48% (N=«71 units) of the units. Forty percent of the elementary school 
units and 55% of the middle school units relate to this outcome area. Less 
than 21% (N«35 units) relate to work seeking and work getting skills, placement 
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Table 1 



RELATIONSHIPS BETWEEN STUDENT OUTCOME STATEMENTS 
AND CAREEH EDUCATION CURRICULUM UNITS 



Student Outcome Statements 



Eleaentary School 
Level 



Middle School 
Level 



Totals 



Number Percent of Number Percent of Number Percent o( 

Units of Units 



of 
Units 



Units 



of 
Units 



Units 



* I. Increased Self 36 40Z 

Awareness 

II. Basic Academic and 76 83Z 

Vocational Skills 

III. Avareness of Work 67 73Z 

Values-Desire to work 

* IV. Awareness and Knowledge 65 712 

of World of Work 

ft V. Competency in Gareer 66 72Z 

Decision Making 

. VI. Good Work Habits 27 30Z 

VII. Work Seeking and 6 7Z 

Work Getting Skills 

VIII. Placed in Occupation 0 OZ 

or Further Education 

IX. Awareness of Means of ■ 
Continued Education 

Total number of units at elementary level " 91 
Total number of units at middle level - 64 
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in an occupation cr further education, or awareness of means of continued 
education. The student outcome statements noted with an asterisk require 
third-party evaluation as specified In the federal guidelines. 

Interest In performing this task was also for the purpose of determining 
the emphasis of the evaluation. ETS maintains that it is imperative that 
the evaluation of student outcomes focus on those instructional goals judged 
by the teachers a being of Importance. Since ETS had previously determined 
the relationship between CEIS Instructional goals and student outcome state- 
ments in the federal guidelines (see Figure 1) a decision could be now made 
as to the focus of the student evaluation. In order to meet the require- 
ments of the evaluation of the federal guidelines and be responsive to pro- 
ject instructional goals ETS plans to concentrate on evaluating the following 
student outcome statements: I. Increased Self Awareness: II. Basic Academic 
and Vocational Skills: IV. Awareness and Knowledge of the World of Work; 
and V. Competency in Career Decision Making. These student outcome state- 
ments are related to the following CEIS project instructional goals: Aware- 
ness Phase, 2. Pupil's Self Awareness, 3. Expand the Occupational Awareness 
and Aspirations, and 4. Improve Performance by Unifying Subjects Around Career 
Development; and Exploration Phase, 5. Experiences to Assist in Evaluating 
Interests, Abilities, Values and Needs as they Relate to Occupational Roles, 
6. Opportunities for Further Exploration of Selected Occupational Clusters 
Leading to Selection for Indepth Exploration, and 7. Improve Performance 
in Subject Areas by Unifying Around Career Development. 

Limitations of Evaluation 

Since career education activities not conducted under the aegis of the 
the CEIS Project are prevalent in the Newark School District it is Important 
that ETS establish some boundaries or limitations of its evaluation efforts 
in order to focus on those career education activities directly related to 
the project. Although the pervasiveness of career education activities and 
the unknown influence that these activities have on the CEIS project make 

this is a difficult task, ETS Intends to limit its evaluation to those career 
education activities which were funded and brought about by the CEIS project. 
Thus, career education activities which were present before the CEIS project 
was operating, such as vocational education, job placement, etc. , will not be 
part of the evaluation. However, job placement rates and other indices might 
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be used to assess the impact of the CEIS project. 
Student and Nonstudent Evaluation Groups 

The draft federal guidelines prepared by Developmental Associatea, Inc. 
are intended to aid career education programs in conducting third-party 
evaluations and are not intended to hinder either program management or 
evaluation. The guidelines focus on product evaluation for elementary through 
secondary school students, with little attention being given to process evalu- 
ation or evaluation of nonstudent groups. These two areas are a major concern 
to the CEIS project. 

Nonetheless, an attempt was made to conform to the federal guidelines 
where appropriate to the CEIS project. Since only one of the CEIS project 
goals deals explicitly with student groups it was important to identify other 
groups which are involved with the project. Table 2 lists targeted student 
and nonstudent groups involved in the CEIS project. 

Table 2 

Targeted Student and Nonstudent Groups 
Involved in the CEIS Project 



Student Groups 



Elementary Schools 



- Brookside 

- Cobbs 

- Leasure 

- Maclary 

- McVey 

- Wilson 
Middle Schools 

- Central 

- Gauger 

- Ogletown 



Nonstudent Groups 



Counselors 

Administrators 

Advisory Council Members 

Teachers 
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Treatment Group / Outcome Area Tables 



The Treatment Group / Outcome Area Table ( TG/OA ) plots and classi- 
fies each of the. 155 curriculum units developed by the CEIS project. Since 
each one of the units deals with some career education concept and the de- 
cision to present each unit is made by the individual each one of the units 
can be considered a treatment. In essence, the TG/OA tables contain 155 
treatments. The TG/OA tables are presented in Appendix B. 

The TG/OA tables contain the following information for each of the schools 
involved in the CEIS project: 

(1) Teacher presenting the unit. 

(2) Grade level for each unit. 

(3) Title of each unit. 

(4) What instructional goals each unit covers. 

(5) What student outcome statements each unit covers (i.e., statements 
listed in federal guidelines. 

Because curriculum units are not currently being taught at the secondary 
level the TG/OA tables were limited to grades K through 8. 

The task of completing the TG/OA tables demonstrates one of the funda- 
mental weaknesses of the guidelines. The nine outcome statements listed in 
the guidelines all relate to student outcomes. Consequently, there is no sys- 
tem for classifying and plotting outcomes for non-student groups. 

Outcome Question / Treatment Group Matrix 

The Outcome Question/Treatment Group Matrix (OQ/TGM) identifies the 
specific sub-objectives of the Guidelines' design which are appropriate to 
each of the student groups. Since each of the Guidelines' objectives relate 
to student objectives, this matrix is completed only for student groups. Tea- 
chers involved in the CEIS project provided the initial input for the OQ/TGM. 

The OQ/TGH appears in -^pendix C. 

As stated previously, the purpose of the evaluation is threefold: (1) to pro- 
vide feedback to the project management for decision making; (2) to assess the 
extent to which the goals of the project have been achieved; and (3) to assess 
the extent to which project related activities have produced side effects. Each 
one of these purposes is further discussed in subsequent sections of the eval- 
uation under the headings of process evaluation ^ product evaluation and side 
effects evaluation. 
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PROCESS EVALUATION 



Purpose of Process Evaluation 

Process evaluation provides information to project directors for the pur- 
pose of making decisions about the day-to-day operation of a project. Process 
evaluation has three primary functions. The first function is to provide feed- 
back to the project directors so that they can monitor the operation of the 
project and detect potential problems in the developmental and implementation 
aspects of the project before they become acute. The second function has to 
do with helping project directors make day-to-day decisions during the course 
of the project. Lastly, process evaluation serves the purpose of recording 
the occurrence of. events through the life of the project. 

Identification of Process Activities 

An effective process evaluation is dependent upon the identification of 
explicitly stated activities designed to acnieve the project's stated goals 
and objectives. In this regard, the CEIS project direction staff is to be 
commended in their preparation of a management plan for the life of the pro- 
ject (see Appendix A). The CEIS management plan covers processes associated 
with all of the intended target groups and will be used as the basic docu- 
ment for formulating the process evaluation. 

Classification of Activities 

In order to provide a conceptual frame of reference for the process eval- 
uation, a three dimensional model is presented in Figure 2- This model de- 
fines appropriate process activities for the project according to CEIS pro- 
ject goals, targeted groups and activities. The purpose of the model is to 
aid in the selection of appropriate activities for process evaluation. Since 
process evaluation is most beneficial if it focuses on the expressed needs of 
project management , the selection of activities to be included for process eval- 
uation will be made in cooperation with the, CEIS project director. 

Identification of Outcomes 

In addition to specifying broad and specific objectives for the student 
target groups (i.e., elementary and middle school students) the CEIS project 
director has also specified objectives for nonstudent target groups such as 

30 



-10- 



advisory committee members, counsellors, etc. Although these have not been 
spelled out completely for all groups, the project is to be commended for their 
work on this task. These objectives are included as part of the project's 
management plan (see Appendix A). 

Process Evaluation Questions 

Before one can expect changes in student behavior, individuals who come in 
contact with students must in some way, change their behavior. Nonstudent 
groups including teachers, counselors, administrators and to some extent, em- 
ployers, must change their behavior as a prerequisite for student behavioral 
change. In order to bring about behavioral changes in nonstudent groups, three 
elements must be present: (1) knowledge, (2) attitude, and (3) resources (Hardy, 
19 75). Ihus an effective evaluation design can be based, in part, on the de- 
termination of the knowledges, attitudes, and resources of nonstudent groups. 
Examples of evaluation questions in each of these areas follows. 

Knowledge 

Do teachers know the goals of career education for their res- 
pective grade level? 

Attitude 

Do principals view career education as a worthwhile concept? 

Resources 

Do teachers have access to relevant career education resource 
materials? 

Actions 

Are counselors using career education materials as part- of the 
guidance process? 

Answers to these and similar questions will provide valuable feedback to 
the project director about the strengths and weaknesses of the project as 
well as insight into possible explanations for the presence or absence of stu- 
dent achievement. 

Steps in Process Evaluation 

A list of the steps to be used in systematically conducting process eval- 
uation is shown in Table 3 and a form has been prepared which will be used in 
recording and formulating the evaluation strategy (see Figure 3). 

The following is a description of the elements of the Process Evaluation 
Form (PEF). 
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Table 3 

Steps for Process Evaluation 

1. Evaluator reviews CEIS management plan with project directors. 

2. Evaluator selects activities to be evaluated with approval of project 
directors. 

3. Evaluator prepares Process Evaluation Description Form (PEDF) for each 
activity to be evaluated. 

4. Evaluator prepares calendar for data collection activities and coordin- 
ates data collection with project directors and target groups. 

5. Data collected according to PEDF. 

6. Data analyzed according to PEDF. 

7. Report of PEDF prepared by Evaluator to be Included in next proj<?ct 
quarterly report. 
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(1) Identification of Target Group 

Since the CEIS project has a large number of target groups which will 
receive services from the project It Is Imperative that each group 
be explicitly stated and Identified. Many of the process evaluation 
activities are aimed at groups of people (I.e., teachers, counselors, 
administrators, etc.) and may require sampling as contrasted to cen- 
sus evaluation. If a sample Is used, the size and method of drawing 
the sample will be reported. 

(2) Identification of Critical Dates 

The critical dates include the start and completion dates for all pro- 
ject related activities, important due dates for the submission of 
project documents and reports ^and any significant interim dates. 
These dates are already part of the CEIS project management plan and 
will be used by ETS to determine when would be most appropriate to 
evaluate the Impact of a particular activity. 

(3) ^ Method of Assessment 

The methods for assessment far process evaluation are more varied 
and less rigorous than those used for student product evaluation. 
Assessment methods will most likely Include the use of interviews, 
questionnaires, and review of planning documents, project records and others* 

(4) Type of Analysis 

The type of analysis to be used in conjunction with process evalu- 
ation may range from a trip report of the evaluator to the use of 
multlgroup descriptive and inferential statistical analyses. The 
type of analyses will be reported on the PEF, (Process Evaluation 
Form) . 

(5) Sample Items 

In order to provide consistency between evaluators and to aid in inter- 
preting the data, sample items will be provided on the Process Eval- 
uation Form* 

The primary source of Information for completing the PEF will be the CEIS 
management plan. Once the PEF and the evaluation has been completed, a re- 
port will be written including a copy of the PEF and a description of the re- 
sults, conclusions and recommendation of the particular process evaluation. 
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The identification and selection of activities to be Process Evaluated 
will be based on the CEIS management plan* Due to time and fiscal con- 
straints, it will be impossible to evaluate all of the activities included 
in the management plan. In order to provide as wide a coverage as possible, 
ETS plans to make decisions in cooperation with the project staff in order to 
insure that the evaluations serve the expressed needs of the staff and also 
in the interest of evaluating as many of the cells in the Process Evaluation 
model as possible (see Figure 2). Every attempt will be made to conduct 
at least one evaluation activity in 50% of the model's cells before January. 1, 
1976. An example of possible entries for the PEF is shown in Figure 4. 
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PRODUCT EVALUATION 



Purpose of Product Evaluation 

Product evaluation focuses on the degree towhlch program goals have 
been achieved ( Anderson, Ball and Murphy, 1975 ). For the purposes of 
the CEIS project, product evaluation Is defined as the measurement of 
student outcomes In those target groups receiving somt direct career educa- 
tion treatments. This particular context- treatment refers to each one of the 
career education units developed by teachers for use In the CEIS project. 

Identification and Selection of Student Group 

The target student groups to be Included In product evaluation for the 
period September 1, 1975 to December 31, 1975 are limited to elementary 
and middle schools Involved In the CEIS project. As was reported earlier, 
no evidence can be cited which supports the Involvement of secondary school 
students In career education treatments under the aegis of the CEIS project, 
therefore, this student group will not be Included in product evaluation ac- 
tivities during the time frame stated above. In the Newark School District, 
secondary school includes grades 9 through 12. 

As was stated previously, the determination of classes to be included in 
the product evaluation will be made in early September, 1975 when information 
is available about what classes will be involved in the CEIS project during 
the first half of school year 1975-76. Due to the absence of this information 
the number of students or classes to be included in the product evaluation 
cannot be determined at this time. Also, decisions about the type of sampling 
design to be used to draw the sample will have to wait until more information 
is available about the number and levels of grades involved in the CEIS pro- 
ject. 

Evaluation Design 

ETS plans to implement a pretest-post test comparison group design based 
upon a sample of intact classes within grade levels. In operationalizing the 
design, ETS plans to pretest and post test the classes immediately before and 
immediately after receiving the treatment ( treatment is defined as the 
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implementation of a career education curriuclum unit in a given class ) in 
order to increase the probability of achieving significant differences be- 
tween the two measure. Furthermore, by avoiding the more typical pattern of 
pretesting in the fall and post testing in the Spring, ETS will be able to i- 
dentify early in the life of product evaluating which instruments seem to be 
most sensitive to the treatments. This information should ^i....^nce the likeli- 
hood of measuring the significant effects of the units. 

Identification and Selection of Comparison Groups 

ETS plans to identify comparison groups using a two step process. In 
the first step, ETS plans to identify those intact classes not reported as 
being involved in the CEIS project. Information will be gathered on each 
class's possible unintended exposure to the CEIS project or career education 
concepts comparable to those found in CEIS curriculum units. The classes 
will then be ranked according to the amount or level of exposure to career ed- 
ucation concepts. The second step will involve selecting those comparison 
group classes which have the lowest exposure rate. ETS plans to select the 
same number of classes in the comparison group as those in the treatment or 
experimental group. For example, Leasure Elementary School had a total of 
five 4th grade classes, two of which were receiving CEIS treatment, ETS would 
select those two out of the remaining three which has the lowest exposure 
rate to form the comparison group. 

Identification and Selection of Instruments 

The identification and selection of instruments to be used for measuring 
the attainment of career education competencies is a challenging task. Since 
product evaluation will focus on grades kindergarten through eight, many of 
the more promising instruments will not be able to be administered because 
they have been developed for use with secondary school students. Also, many 
of the instruments appropriate for administration at the elementary and high 
school level are poorly designed and have relatively little statistical data 
to support their underlying assumptions. (ETS is currently reviewing avail- 
able career education instruments for possible use in the evaluation.) 

Another problem is the interpretation of the subscales on published tests 
and their relationship to the CEIS project goals and objectives. Although 
ETS has reached consensus with the CEIS project staff on the relationship 
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between CEIS goals and student outcome statements listed in the federal 
Guidelines (see Figure 1 ) differences are bound to exist between what a 
test purports to measure and project goals and objectives. 

In approaching this problem, ETS is currently surveying available 
career education Instruments for use in the evaluation. After this survey 
has been completed and a pool of potential instruments has been identified, 
ETS plans to review these instruments with the CEIS project staff and tea- 
chers to make the finfil selection. ETS also plans to work in cooperation 
with the Newark, Delaware director of testing in this selection process. 

Once the instruments have been selected and teachers have indicated 
what curriculum units will be taught during the first half of the school 
year, ETS will prepare a testing schedule. The testing schedule will include 
the name or section of the test to be administered, test dates for pretesting 
and post testing, and the names of the schools and teachers in the treatment 
and comparison groups. Hie following is a list of some of the guiding prin- 
ciples to be used in conducting the product evaluation: 

(1) A pretest-post test comparison group will be sought for all instruments. 

(2) Total testing time for students will not exceed two hours. 

(3) Project staff, teachers and the Newark director of testing will be in- 
volved in reviewing instruments. 

(4) Teachers will be notified two weeks in advance of testing data. 

(5) Testing dates will be checked with the director of testing to avoid 
possible conflicts with other testing programs. 

Data Analysis 

Detailed plans for data analysis of the results of the product evaluation 
will be made after reviewing the raw data. It is planned, however, that dis- 
tributions, means and standard deviations will be generated for each group 
tested. The final determination of the appropriate statistic to use when com- 
paring differences between pretests and post tests and between treatment and 
comparison groups will have to wait until the data has been closely inspected. 
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SIDE EFFECTS EVALUATION 



Purpose of Side Effects Evaluation 

EvaluatorJ? should try to assess not only the Intended outcomes of an 
educational project but also unintended outcomes. The assessment of unin- 
tended outcomes Is referred to as side effects evaluation. The CEIS project 
Involves the Interaction of a wide array of target groups and as such the like- 
lihood of unintended outcomes seems highly probable. For example, the CEIS 
advisory committee Is made up of representatives of governmental, education- 
al and community agencies and business and Industry working together to real- 
ize the project goals. This Interaction might result In a positive side 
effect such as Increasing the availability of cooperative education work 
stations or changing the attitudes of businesses toward on-site class visita- 
tions. Conversely, it is also possible that this interaction could be viewed 
by the council members as being a waste of time and might result in some mem- 
bers sharing a negative view about the. CEIS project specifically and career 
education in general. 

Identification and Selection of Side Effects to be Evaluated 

Two of the major problems encountered in evaluating side effects are; 
Which potential side effects should be evaluated? Who makes the decisions? 
( Anderson, Ball and Murphy, 1975 ) The problem of deciding which side effects 
to study out of a potentially unlimited number is not an easy problem to re- 
solve. In an attempt to focus this decision making process, ETS plans to 
limit its investigation of side effects to those concerned with the project 
management and the CEIS advisory committee. In terms of answering the sec- 
ond question, ETS plans to make this decision in cooperation with the CEIS pro- 
ject directors. 

Similar to the elements of the process evaluation model, ETS has pre- 
pared a list of steps for side affects evaluation which is illustrated in 
Table 4. A form for ecording side effects evaluation ( SEEF ) has also 
been prepared and is displayed in Figure 5. An example of possible entries 
for the SEEF is shown in Figure 6. 
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Table 4 



Steps for Side Effects Evaluation 

1. Evaluator Identifies possible side effects with project directors. 

2. Evaluator determines which side effects to evaluate with approval of 
project director. 

3. Evaluator prepares Side Effects Evaluation Form (SEEF). for each side 
effect to be evaluated. 

4. Evaluator prepares calendar for data collection activities and coordinates 
data collection with project directors and target groups. 

5. Data collected according to SEEF. 

6. Data analysed.- according to SEEF. 

7. Report of SEEF prepared by Evaluator to be included in next project 
quarterly report. 
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CEIS Management Plan 
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Appendix B 



Treatment Group / Outcome Area Tables 
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NEWARK SCHOOL DISTRICT CAREER EDUCATION PROJECT 
PROGRAM GOALS — AWARENESS PHASE (K-5) 

— to develop in pupils positive attitudes about the personal 
and social significance of work; 

— to develop each pupil's self awareness ; 

—to develop and expand the occupational awareness and occupational 
aspirations of the pupils'; 



— to Improve overall pupil performance by unifying and focusing 
basic subjects around a career development theme. 



61 



CO 



J-3 



ERIC 




NEWARK SCHOOL DISTRICT CAREER EDUCATION PROJECT 
PROGRAM GOALS — EXPLORATION PHASE (6 - 8 ) 



5. — to provide experiences for students to^ assist them ±n evaluat Ing 

their Interests , abilities , values and needs as they relate to 
occupational roles; 

6. — to provide students with opportunities for further and more detailed 

exploration of selected occupational clusters , leading to the tenta- 
tive selection of a particular cluster for Indepth exploration ; 

7. — to Improve the performance of students In basic subject areas by 

making the subject matter more meaningful and relevant through 
unifying and focusing It around a career development theme. 



1 



72 



-46- 



I 



Appendix C 



Outcome Question / Treatment Group Matrix 



EKLC 



-53- 



POWELL 



MOODY 



LECK 



O'NEILL 



O'NEILL 



O'NEILL 



WEAVER 



SHAY 



RICKERMAN 



MOORE 



COX 



a. 

O 
O 



c 

0) 
H 



C 

o 

AJ 
U) 
0) 

a 
o 
u 
u 

O 



cn 
w 

d 

u 



tn 
dJ 
u 

c 

M 











tn 
















dJ 








AJ 






OJ 


•H 








tn 






-n 


ij 






-Q 








•H 


•H 






■H 






OJ 








0) 










U 






tn 


U 


ij 




13 


tn 


•8 




nJ 


tn 


d 




0) 






dJ 


dJ 






P 








t-i 


T3 






U 




AJ 




u 




AJ 




c 


o 


d 




C 


O 


d 






AJ 


re 


if 




AJ 


re 




tn 


>^ 


u 


c 


tn 


>^ 








AJ 






ij 


AJ 






c 


•H 


u 


4- 


d 


•H 


u 






iH 




rt 


0) 


.H 




tn 




•H 


d 








d 


OJ 




XJ 


:j 












AJ 


nJ 


o 


f 


u 


o 




V] 








tn 


















u 


> 


0) 


•H 


•H 






•H 








OJ 


OJ 




> 








Ha 


55 




Ha 


5 


x: 
ij 


an 









1 


al 


1 


















O 


d 




















tn 


o 


r-l 




















•H 


IM 










iH tn 


2 ^ 




(U 


iJ 


AJ 


C 


AJ 










S o 




tn 




nJ 




d 






tn 


nJ iH 






nl 




u 




OJ 


•H 




nl 




>> 




dJ 






tn 


&• 






OJ 


OJ ^ 


m tn 










OJ 




dj 






d tn 


iH OJ 




u 


d 


0) 


u 


o 






u 




a* T3 




d 


o 




u 








d 




tn 3 






•H 




o 












•H AJ 






AJ 


U 




> 




tn 




U4 u 






tn 


•H 


•H 




OJ 






tn 


O 


AJ 




AJ 


d 


a 




T3 


o 




AJ 




tn AJ 




d 


to 


o 






•H 


•H 


d 


iH QJ 


u nj 


tn 


(U 


o 


d 


u 


U 


tn 






(u a 


c: 


OJ 




u 


o 


d 


•H 




CO 




> d 




> 




OJ 


u 


AJ 


OJ 


OQ 




d 




T3 > 




AJ 




(U 


r-l 


x: 






AJ 






OJ 


tn 






d 


AJ 


T3 




tn 




AJ iJ 


tn 




u 




u 




(U 


d 




M d 


tn -H 






•H 


.H 






tn 


o 






tn 


dJ 


> 


OJ 


(TJ 




u 


nl 


•H 


> 


OJ (U 


o o 








•H 




d 


dJ 


AJ 






O a* 


AJ 






U 






t-i 


nJ 




















o 


u 






d 














d 


o 








Q 










M 


> 




















M 






















M 









80 

-54- 



POWELL 



MOODY 



LECK 



O'NEILL 



O'NEILL 



O'NEILL 



WEAVER 



X 



X 



X 



SHAY 



RICKERMAN 



MOORE 



COX 




OJ iH 

(TJ d 

dj dj o 

u ,cu u 

•H u 

u-i .H 

o c 



tn 

d .H § 

dJ 0) Q 

> o 
3 (U u 

AJ iH 

cn iH 

u d - 

0) -H u-i H 

> dJ dJ 'H 

PC u 3 tn 



dj H 

cn ^ 

nJ M 
dJ QJ 

u c 

u dJ 
C ^ 

cn o 
u 

C H 
dJ 0) 



cn 

u 

dJ 

> dJ 
5d 



a 



dJ 



I 

o 
u 
a* 

c 
o 

AJ c^. 

nJ cn 

B M 

u iH 

o "H 

u-i ^ 

C cn 

60 

cn 

dJ cn 

3 25 



T3 .H 

dJ .H 

cn nJ 

d) dJ 

u c 

a dJ 



60 



cn o 

AJ 

C iH 

dJ dJ 
> 



5d 



c 

a 
I 

c 
o 

cn 

•H 

u 
dJ 

iH cn 
d iH 

dJ -H 
cn ^ 
3 cn 



n3 iH 

dJ iH 

cn cd 

ni M 

dJ dJ 

n c 

u dJ 

C 60 

cn o 

AJ 

C iH 

dJ dJ 

Td > 

d d) 

AJ iH 

cn 

dJ 
> 



C 
o 
cn 

dJ 
dJ 

AJ 



u 

•H 

dJ 

PC AJ 



iH cn 

d iH 

u-i .H 

dJ -H 

cn M 

3 cn 



X 



X 



u 
o 

:5 

O 

cn 
cn 
dJ 
c 

dJ ■ 
cn 

nJ cn 
dJ dJ 

M > 



d) AJ 

cn 

dJ o 

u 
c 



cn 

C 
dJ 

3 
AJ 

cn 

u 

dJ 'H 
> dJ 

PC AJ 



•id 
u 
o 

cn 
o 

H 

> 



cn 

(/) dJ 
dJ d 



81 



-55- 




82 

-56- 




ERIC 



83 

-57- 



h4 
O 
O 



W 

PP 
O 

u 



POWELL 



MOODY 



LECK 



O'NEILL 



O'NEILL 



O'NEILL 



1-3 
o 
o 



CO 

O 

§ 



WEAVER 



SHAY 



X 



RICKERHANf 



MOORE 



COX 



o 
o 

c 

e 
•J 

0) 

H 



U I M-l 
■H O O 

^ U 

W *4-f O 

(tJ -H en 



U (U 
C T3 



5-1 
0) 

OJ c^- 

(TJ g 

O OJ 
rH 

OJ ^ 

> O 

—I U 
O 



QJ O t 

W U 

0) d 

0) AJ > 
C/} 

(tj 0) ctj 

OJ c 



tn 
rJ 
o 

OJ 
OJ 

e 

o 
u 
ij 

O 



60 



o 

AJ 

>^ 
AJ 



6 
I 

C 

o 



■H 0) AJ 
AJ T3 

o o 

AJ 



O 



AJ 

o 
u 

00 



■H QJ 
OJ ^ 

AJ O 
AJ 

QJ CO 
CO Pu CO 
(T) OJ 
U 
CO 



0) 

O 
AJ 



O 
AJ 
U 
OJ eg 



(J 

c 

AJ 



> -H AJ 



^1 to 
O -H 

-Si 



AJ 



0) H iJ 
> -H C 

jQ QJ 



d 

o cj 

CO 

QJ -H 

p. QJ 

AJ AJ 

c 

QJ U-l 

e o 



CO 



AJ tM 

C O AJ 



OJ 

3 



OJ t3 
£10 



4J 

CO OJ 



c 

o ^ 



I 

c 

rH 

u 

OJ 
OJ 

(tJ 
u 

C 
■H 

T3 
OJ 
U 
QJ 

CO 00 

d d 
o •H 
u d 



OJ OJ 
QJ 

u 
u 



AJ 

d 

OJ d 

CO > -H 

AJ rH ^ 

O CD 

> S 

d I 
H d 
o 

rl 
CO 

OJ -H -H 

> AJ a 

U 0) 



d 

OJ 
AJ 

CO 0) 



-58- 

81 



H 



CD 
H 

M 

s 

H 

o 

M 
H 

o 



POWELL 



MOODY 



LECK 



O'NEILL 



O'NEILL 



O'NEILL 




w 
c 
o 

•H 
AJ 

cn 
dJ 

0) 

6 
o 
u 
u 

o 



o 
u 



cn 






AJ 






•H 


JD 




JO 


tT3 








> 




tn 


•H 




u 


AJ 






O 






dJ 


O 




tM 


13: 




tM 




u 






tn 










> 


dJ 




o 


u 


o 


u 


< 




a 






a 






M 


< 










> 







I 

tn 

T3 ^ 



O T3 



T3 

u 

tn dJ 

^2 



o 

AJ C^* 

cn 
c 
o 

•H 
AJ 



tn 

0) J-i 

U CT] AJ 

O 5 -H 

a O iH 

AJ (TJ 

(TJ 3 

(U a* 

> d u-i 

AJ o 

AJ tn 

tn AJ AJ 



dj u 



> 

AJ U 
•H 

tn dJ 
o 



tn 

(TJ 

§ 

4J 



dJ 

c «9 



> 

dJ Q) 

AJ O 

AJ O 
4J 

dJ (t3 
> > 



a 



tn 
dJ 
u 

o ^ 

tn M 

Q> o 

n :j 

C 60 

a -H 

tn 

•rl 

c a 

cn a 
o 

iH u o 

(Tl U CO 

"AJ (Tl ^ 

c tn 

oj c 

0 XJ 



tn 



u ^ <\} B 
o ^ X o 
a o "tJ CJ 



AJ 

u 
o 

t3 U-l t3 

AJ >. S 
•H AJ 
AJ .H 
4J iH 

> 
•H 
AJ 



U 
O 
•H 

> 
(TJ 

tn 

C dJ 

O J3 

a 

tn tn c 

o d) 5 

u o 



dJ 
tn 
o 

a 

•H 
I 

IM 
.H 

dJ 
tn 

tM 

O c 



tn 
tn 

AJ 



H O 



EKLC 



85 

-59- 



POWELL 



MOODY 



LECX 



O'NEILL 



O'NEILL 



O'NEILL 



WEAVER 



SHAY 



RICKERMAN 



MOORE 



COX 




tT3 


1 

c 


C 














AJ 


AJ 






C 








T3 










o 












1 




CTj 


O 






•H 






dJ 


dJ 




AJ 




u 


o 






c 




dJ 






jc: 






dJ 








> 


W) 


T3 


•f^ 




tT3 












tn 






AJ 




<>• 


cn 


T3 


T3 








C 


:j 




u 


t-l 


:S 








(d 












(TJ 


dJ 


C 






W) 


tn 






u 


o 


o 




W) 




d) 




dJ 


w 


d 




dJ 


u 


(TJ 






c 




dj 












c 




M 




AJ 


0) 


0 


0 


(»l 


•r< 






cn 


dJ 


c 














•rl 




U 






o 


•rl 




CJ 








AJ 




o 


tn 




0 


0) 








cn 


d 


O 


U 


M 


AJ 




\~t 


or 






c 


dJ 


•H 


a 




B 




d 




dJ 




•H 


AJ 


o 




CTJ 




•H 


dJ 


O 




dJ 




AJ 






dJ 


•H 






dJ 


rH 








o 




•rl 




U 


U-l ^ 






tn 


tn 






T3 


W 


g 




00 


•rl 


C/J 


>^ 




w 






cn 












u 


8 






GJ 




o 








JJ 


/• — s 




o 


a 


u 


cn 


60 o 




AJ 


u 








W 


T3 










dJ 


•H 




dJ 






C 


rH 


•S ^ 




cn 


o 


•TO 


cn 




AJ 




dJ 








rH 




to 


c 


§ 


dJ 












dJ 


•H 




c 


T3 




0 






•H 




*H 


•H 






•rf 


>, 60 














dJ 












.r> 




•H 
















dJ 


t-l 


AJ 


i2 




tn 


tn 








AJ 




>^ 


•H 


C 


C 


AJ 


cn 


<X •H 


AJ 




u 


dJ 




tn 




«T3 








AJ 








U 






cn 




a AJ 


c 




fd 




cn 


dJ 


AJ 


dJ 


o 


o 


0) 


AJ 






•H 














dJ 




rH 


AJ 


dJ 




cn 










dJ 


dJ 




AJ 




AJ 




dJ 


•H 


dJ 






a 




o 






o 


c 




C3 


o 


> 




c 




c 


•rl 


> 


dJ 


^ o 


d) 








u 


o 


d 




U 


Q) 




ITS 




dJ 


U 


o 






:S 


,CT3 u 


o 




c 






o 


P 




AJ 


to 


ncr 


ork 




u 






u 


a 


Hi 


Pla 




dJ 
nt 
.Q 


in 


how 


pri 










M 


13: 


< 












PP 



















M 
> 



M 



ERIC 



-60- 

86 



o 
o 



DOWNS 



PRICE 



WILSON 



SCHEFFEY 



MORTON 



LEONARD 



H 



2 f- 



BUS SARD 



TURULSKI 



DAVIDSON 



PERRY 



SMITH 



MILLER 



O 



0) 

I I 
^ / 
/ 



<3 



/ 



i/ 



to 
o 

IJ 
f/) 

cu 

o 

u 

O 



cn 
w 
a; 

u 
I 

.H 

0) 



'J 



0) u 

U) u 

nj cn 

u 

a o 

•H iJ 



cn 



•8 



u 

iJ ij 3 
C iH U 
H) .H 



o 



o c 



^3 
OJ U 



o 



'S fi 



ft W 



a 

cn 
> 



(d cn 
ex. 

tn r? 

•H iJ 

ij 

cn -Lj 

nJ cn 
d 0) 
dj dj > 
> 

•H (U 

i-» ' .n 

•H B 

cn dJ 
o 

a. 4J 



o d 
cn o 



T3 

dJ 

cn nJ nJ 

nJ rf u 

dJ tl d 

u d dJ 

d o 



u 

6) o 

o d 

o o 

c- u 

ij >-« dJ 
cn 

dJ -n. iH 

> dJ r! 



d iJ 
•H d 
dJ 

cn 6 
dJ CL 
u o 

o dJ 
> 

iJ (U 

u 

dJ 
u 

S S 



u 

•H 

6 
dJ 

nl 

<^ tn 

r-H 

' ) tH 

cn 



CT3 

d 
o 



dJ 
cn 
nJ 

OJ u 

u (d 
u u 

V-! > 



87 

-61- 



X 

M 

H 



a 

H 

I 

H 

55 
O 
M 

H 

to 

s 

I 

O 

o 




X 



X 



X 





>> 














ase 


ral 










(d 






c: 


C 




u 




o 




c 


W) 


tn 








t-i 
dJ 




cn 


o 










u 




c 


H 


d) 














> 














u 


.H 






cn 










p 




/H 








r-l 


> 




a 








w 








13 





o 

14-1 

o 

tn 
cn 

4» 
G 

S 

:$ 

<: 

d) 

tn 

dJ dJ 

M > 



dJ ' 

dJ u 
cn 

<u o 



o 



o cn 

u o 

C M 

50 03 

O > 
O 
dJ 



U 
H 



O c^. 

cn 

cn oj 
* 3 



dJ 



88 



ERIC 



-62- 



>OWNS 



PRICE 



MILES 



SCHEFFEY 



tr 
I 



MORTON 



LEONARD 



BUSSARD 



TURULSKI 



DAVIDSON 



FERRY 



X 



X 



X 



X 



X 



X 



X 




EKLC 



-63- 



89 





-64- 



9U 



DOWNS 



PRICE 



MILES 



SCHEFFEY 



IT 
I 



MORTON 



LEONARD 



BUS SARD 



TURULSKj 



DAVIDSON 



PERRY 



SMITY 



MILLER 



a, 
o 

M 
O 



<3J 

E3 



s / 



X 



X 



X 



u 


1 


U-l 








o 


o 








r-r 


















u 






dJ 








u 


H 


CO 






p 




6 


rj 






U 


dJ 


cn 


U-l 


o 




rH 


rJ 


•H 


tn 






CJ 


AJ 




> 


O 




c 


dJ 




U 


u 


dJ 


N 


O 


tx 


a 




•H 


tn 




•H 




rH 










•H 


o 


c 


tn 


✓-^ 


AJ 


AJ 




AJ 










c 






a 


d 


o 


O 


u 


o 






AJ 




•H 


r 


d 






AJ 


c 








nJ 


o 


cn 


AJ 
















tn 




H 




o 




> 




AJ 


u-l 


u 








a 


G 






u 






d 











f 

O I 

o 

a 

AJ 



o 



AJ 

tn 0) U-l 
AJ 

GOO 
TJ AJ u-l 

tn 



AJ 



G iH AJ 
> -H C 
r3 ^ G 



AJ 

O 60 

u d 
tn 

rH O 

fd o 

g-s 

tn 

u u 

G -H 
Cu G 

AJ AJ 

a 

G u^ 
6 o 



X 



X 



u 

H G 
G ^ G 

o 

AJ O 
AJ 

G tn 
cl tn 

G P 
iJ o 
cn AJ 
u 

H j:: u^ 

AJ 

tn G 
^ <u x: 

C O AJ 
G 

G n3 
tn G 

rH C 

' ^ G 
O ^ 



AJ 



J3 Cf fd 



a 
p 

G 

G 

p 

G 

a 

G 

P 

G 

tn to 
c c 

O -H 

o c 



X 



X 



X 



W 





P 




G 


G 




r* 


G 




AJ 


ar 






o 




G 






tn 


c 






•H 




G 






p 


AJ 




a 


c 




a 


G 








to 




G 


c 


tn 


> 




XJ 






c 


O 


a 


G 


> 


B 








d 


•H 


a 


AJ 




o 


tn 


G 


•H 




> 


tn 


G 




•H 


> 


AJ 


u 




O 


G 




<vJ 





-65- 



01 




ERIC 



-66- 



DOWNS 



PRICE 



MILES 



SCHEFFEY 



MORTON 



LEONARD 



BUSSARD 



X 



X 



TURULSKI 



DAVIDSON 



PERRY 



SMITH 



MILLER 




X 



X 



tT3 


1 

c 


C 




OJ 


o 






U 


u 






tT3 








U 




iJ 




iJ 


o 


O 




cn 




JD 




C 








o 


OJ 


W) 


















iJ 




tn 


C 






dJ 




O 


w 








ij 




0; 




d 








dJ 










tn 


tn 












iJ 


dJ 


o 


o 


tn 






0 




u 


C 




o 


c 


•H 


u 


Q 




iJ 


tn 



X 















iJ 














ij 












1 




(tJ 


o 




c 










x: 


JQ 








tn 






ij 






















to 


dJ 






tn 


d 




c 




M 




ij 


OJ 


o 


O 


•H 




O 






u 


•H 






tn 




0 


u 


u 






OJ 


rH 




iJ 


o 


d 








iH 






I-l 


o 


B 




w 




tn 


>^ 




tn 




CT3 








iJ 






O 


a 








•H 


XI 


dj 


tw 










•H 






C 


§ 


o 


00 






—J 


•H 




13: 


d 
















•H 


u 






•H 


C 






AJ 


tn 






iJ 




rd 




U 




U 


'H 








tn 


0) 




•H 


JJ 




u 


Td 




O 


> 


0) 


d 




d 














U 


o 


0? 






PC 


iJ 






u 




u 
















d 


O 


















<^ 












M 
M 
















> 

















X 



OJ 

tn 

(TJ 
OJ 
M 
O 



tn 
u 
d 

OJ 

a 

u 

(A 



d XI 

OJ d 

•H 

a* u 

QJ O 

tn bo o 

H d > 



>^ 60 

rH d 

CL iJ 

x: ,ctj u 







x: 


OJ 


OJ 


ij 




«) -a 


•H 








x: 








d 


0) 


tn 


OJ d 




ij 


o 




d 




cu 














3 


3 


ij 


U 3 




tn 


O 


tn 



iJ ^ 
d 0 

(U o 
U PC 

CM <J 



u <u 



j:^ tn 
iJ u 



CT3 ^ tn 0) 



d ^ 

OJ 
XJ 



u 
o 



EKLC 



-67- 

93 






O 












W 


O H 














<>• 






<v 


u 




u 






tn 






a 


a 








U 


d) 






(U 




d cn 








u 




t3 dJ 


a 


d) 




u 


d 


dJ u 


o 






a 


o 












•H 


o 


dJ 








4J 


U iU 


> 




CO 


cn 


•H 




dJ 








a 






U 


.H 






o ^ 




•H 






o 


a u 




V) 






u 


o d 


•H 


ctJ 






(U 


U 4J 


dJ 


pq 




u 




dJ iH 






.H 


cn 




d 


u 










- u 




dJ 






•H 




dJ 


CO 


o 


> 




. J 


o 














dJ 


4-1 


DC 




u d 


d) 




(d 










U 


u 










a 


o 


Q 








H 


> 




















1 


M 





X 











r . 


(0 


dJ 


iH 




as 


fd 


.H 


u 




dJ 


dJ 




M 


d 


CO 


U 


dJ 




d 


e)0 




•H 


M-i 


o 




o 




XJ 






d 


tH 


d) 


dJ 


dJ 






> 




d 


dJ 


d 


4J 






cn 










d 


dJ 






> 


dJ 


dJ 


Ha 


s 


US 



-68- 

94 

o 

ERIC 




95 

-69- 



ERIC 




ERIC 



-70- 

9G 



r 



1^ 

8 
g 

% 

H 



S3 

o 

»-3 



BENSINGER 



WHAYLAND 



WHENN 



JENKINS 



HYLENSKI 



CLARK 



MORRIS 



HART 



X 



X 



X 




a 

•H 

c 

H 
u 

o c 

.H 3 



X 



X 



X 



I 

B 

H 



n) -u ij ij 



u I 
dj u 

U-l (TJ 



0) 

u u-l 
u o 



cn 

> (U 



jz tn 
ij 

u m 
o 

ij tn 

u (n 

u-l u 
u 

ij 3 

C cn 

u u 

o o 



X 



X 



X 



I 

a 
o 

H 

tn 

H 

u 

OJ 

Q 

u 

dJ 
dJ 

a 

-a 
dJ 
cn 
ni 
dJ 
u 
u 
c 



T3 

d) u 

tn o 

nJ cn 

dJ tn 

u ni 
u 

C! o 



dJ 'H 



I 

u 

cn dJ 

dJ d 

•H U-r 

cn .2^ 

cn M M 

o o o 



cn >^ -H 



dJ H 
3 U3 



nJ o 



iJ 
cn 

dJ -H -H 

> dJ dJ 

J? 

tl^ iJ iJ 



dJ -H 

cn nJ 

dJ cx 

•H O 



X 



X 



cn 

Td dJ 4J 

dJ iJ cn 

cn nJ dJ 

dJ 0) dJ 

n n ^ 

o a 

c o -H 



dJ 



nJ cn Td 

iJ cn 

'H dJ nj 

S u cx 



o 
w 

M 

d) 



cn nJ 

iJ iJ c 

dJ r-i 

Td .r^ 

3 JO . 

ij nj cx 

cn 

u u 

dj -H -H 

> dJ dJ 

123 tJ 4J 



P3 



w Td >>, 
dJ d iJ 

^1 

4J ^ OJ 
H M-l 
O O -H 

cn Td Td 

OJ 'H OJ 

iH cu nJ 

> d u 
o 
cn 
w 

CO 



Td nd 



ERIC 



-71- 

97 




-72- 



98 



WALLER 



WALLER 



1-3 
O 

o 

B 



SCOGNA 



MCCAFFREY 



X 



1^ 



1^ 



HOLMES 



COOK 



SARDANA 



RUDISELL 



8 



Q 



H 

8 



REED 



OWENS 



MORRILL 



X 



DEFORREST 



r- X 




to 
cn 

d 

0) 



I 

U-l 



0) 



o 
c 



dJ 

dJ u 

tn u 

to tn 

dJ dJ 

o 

c o 



m 
dJ 

ij 

■H 

•9 



c 

dJ 
u 

cn M 

H 



1) iH 
3 XI 



u 



O E 



d) 0) 

•S 5 



dJ cn 

^ d) 

•H n 

t-i dJ 

U iJ 

cn c 

dJ iH 

o C 

AJ dJ 
u 



3 

> 



b 

u u 
d) d) n3 



55 









1 


al 


1 










O 


c 






dj J-« 






cn 


o 














•H 




<>• 


2 ^ 




dJ 


AJ 


AJ 


c 


AJ 


0 o 




cn 








c 










O 




dJ 






dJ 




d 


cn 


a 


ctJ cn 




(-1 






d) 


a- 


■H OJ 




u 


c 


dJ 


u 


o 






c 


o 




t-i 


.H 


cn 3 






•H 




o 


dJ 


•H AJ 






AJ 


u 




> 






cn 


•H 






dJ 


ij 




AJ 




a 






J) AJ 




c 


& 


o 






AJ (TJ 


cn 


dJ 


O 


c 






c 


dJ 




u 


o 






dJ dJ 


> 




dJ 


u 


AJ 


dJ 


> 


.H 


ij 


(-1 


dJ 


.H 






d) 


cn 








AJ 


iJ iJ 


cn 




(-1 




u 




cn -H 




dJ 


■H 


.H 




dJ 




d) 




dJ 




Td 


u 


o o 












C 




AJ 






U 










Q 

























iH cn 






dJ 


















U 1-1 


a 




dJ 


dJ ^ 


dj 






C cn 






u 


dJ 






in 


•^■^ 




cn 




U-« CJ 






cn 


O 


u 




AJ 




•H 


•H 


c 


iH dJ 


cn 




dJ 






CO 


Td 




pq 






dJ d 




.H 


ij 




Td 




cn 




dJ 


C 






cn 


o 


dJ 




nJ 




> 


dJ dJ 


re 


at 


Ha 




o 


o 






c 


o 






M 


> 


< 




M 








M 









ERIC 



99 




ERIC 



-74- 

100 



WALLER 



X 



WALLER 



X 



SCOGNA 



MCCAFFREY 



HOLMES 



COOK 



SARD ANA 



cc 



RUDISELL 



X 



REED 



a 



OWENS 



X 



MORRILL 



X 



DEFORREST 



X 








0) 


rd 




ow- 






d 


1 


I 


-a 




0) 


















0) 








00 




o 














M 








tT3 






o 








to 


t-i 




Td 




















(TJ 






Td 




d 




tn 










0) 


bO 






•H 




w 


o 


tn 


tn 


O 








0) 




0 


nj 






:^ 


0) 










u 


t-l 






a 




o 


tn 






tn 




d 




tn 


tn 










tn 






tn 




•H 


0) 


dJ 


0) 


U-l 








o 






0) 








•H 


0 




0) 




tx 






d 




tn 


Td 


ij 


XJ 












4-1 




0) 




ij 


dJ 






cn 






c 


tn 






V4 


o 


d 






H 


0) 




w 




ij 


(tJ 


B 


(TJ 




0) 










U 


dJ 




d 












o 


u 






o 


o 


U 


0) 


0) 












o 








O 




> 


C 






























O 


tn 








ij 










XJ 




0) 






u 


B 


Q) 


d 




tn 




tn 


•H 




cn 


•§ 


a) 






U 


a) 










U) 


•H 


CO 




> 


(1) 


0) 








dJ 




o 


o 




CJ 






-5 


rd 




0) 












a 








iJ 






u 










u 
















In 


Pa 








In 


le 





















cn 




-a 




>> 




Td 




Td 


1 


1 


(0 


ij 




Td 










dJ 


U-l 






d 




0) 


ij 








tn 




-a 


. tn 


dJ 


a 


tn 


d 


o 


a. 








d 


dJ 


t-i 


d 




dJ 












0 




dJ 




0) 




§ 


3 






y^ 


u 






u 










o 


o 




JJ 




Td 


u 


O 




Td 




d 






tn 






d 










•H 


dJ 












dJ 


O 








W) 


o 


OJ 














tn 






U-l 


x: 




tn 


Td 




Td 




4-} 


d) 




•H 


4-1 






dJ 




•H 




d 




c 


rH 






d 




JJ 










•H 








0) 


15 


d 


a. 






§ 




Td 






Td 


0 


dJ 








d 


tn 




Td 


















dJ 




d) 


o 


ij 




dJ 


o 




tn 




u 




ij 




tn 




u 








d 


cn 


(d 










cn 




dJ 




d) 


d 




dJ ^ 


dJ 


•H 




d) 






d) 




o 


u 


a u 


> 


dJ 


C* 


a. 








0) 




o 


o 






'J 




O 










tn 


4J ^ 






u 


4J 




PQ 























-75- 
101 



WALLER 



WALLER 



X 
H 



H 



H 

\ 

O 
M 
H 
CO 

o 

a 



8 
S 

CO 

g 



i 



1-3 
O 
O 

g 

h-3 



SCOGNA 



MCCAFFREY 



HOLMES 



COOK 



a 



SARDANA 



RUDISELt 



REED 



OWENS 



MORRILL 



DE FORREST 




oc 



a 



X 



X 



X 



X 



X 



X 







AJ 










o 


1 








dJ 


























0) 


0) 




cn 




tn 














u 


AJ 




dJ 
















cn 




o 


o 


u 








0) 


cn 


s 






60 


u 






cn 




o 






u 


dJ 


AJ 


cn 






.H 


o 


cn 




0) 






Qi 






o 




0 










0 








AJ 


a 




cn 




d 


cn 






>-< 










•H 


iJ 




a 


> 


0) 


U 




0 




,1:: 


O 


0 


•H 


a: 




O4 


1^ 


AJ 



fx3 



X 



I 

d 

o 

H 
03 
tH 
O 

0) 
OJ 



AJ i-l 

•H Td 

OJ o c 

cn o <t3 
cn 



O AJ 



cn 



AJ 4J ^ 

cn OJ .H 

H d 

-« 3 JQ 5 

aj o 



19 



u u 

- 60 > (U (U 

<U d CTJ J2 J2 

M -H W AJ AJ 

d 'f^ . 

M s <; 



I 

o 

cn OJ 
u 

(U d 
.H AJ 

•H u^ 
cn jii 
cn ^ M 
000 
cx ^ 

AJ 

d n3 

AJ (U -H 

•H cn nJ 

cn D* § 
d 

o d 

•H -H O 
AJ 

«t3 cn 

AJ cn -H 

•H CU « 



OJ 

0) AJ 

cn «^ 

CT3 .H 
0) OJ 



M AJ 



cn X 

AJ AJ 

d -H 

(U iH 

AJ "3 

cn 

> (U 



•H 

cn a.-H 

AJ OJ C^. 

cn u^ cn 

(U O AJ OJ 

OJ cn n3 >s 

d 3 cn 
I 

^ QJ 
U 

O O -H 

d AJ :j H 

o 

cn cn Td Td 

■H OJ 

nj 



1^ 



U 0) 



tri d -H 



n > 

OJ Td Td 

5 g s 



-76- 



ERIC 



102 




-77- 

103 



8 

u 

Q 
Q 
M 

Pi 
u 
o 



WALLER 



WALLER 



SCXDGNA 



MCCAFFREY 



HOLMES 



a 



I 



X 



COOK 



SARDANA 



RUDISELL 



S 

a 

Q 
w 
X 



REED 



a 



OWENS 



MORRILL 



DEFORREST 




o 



cn 










>^ 


T-l 




l-f 


•9 




dJ 
> 




W 


•H 






4J 




a 


U 




dJ 


dJ 


O 




M-l 






M-l 






dJ 




tn 










> 






o 


M 


o 




<: 




a 






B 






M 


< 










> 







X 



I 

JJ 

I 1-1 

Oi cn 

n3 ^ 
o 

dJ 5 
o 

B dJ 

T-f 

cn M 

s 

o 

4-» tn 
tn dJ d 
•H o 

dJ ^ 1-1 

M (D 4-> 



tn 

S 1-1 ^ 

O iH n 

•u nJ o 

dJ a* 
dJ XJ nj 

nJ 4J o o 

jc: 1-1 •«-> 

4j tn 

JJ JJ cj 

nJ o 

dJ 1-1 



tn 



dJ dJ cj 
3 1-1 



tn 

tn dJ 
o o jz 



o iH 



tn 
Td 
dJ M 

§ § 

ij 
dJ 

dJ 

> 3 
1-1 
tn •»-» 
d 

dJ dJ 
n3 > 
3 -H 





tn 








dJ 






1 


u 






d 


u 






o 










o 






1-1 


tn 






> 


dJ 


o 




d 


u 






0) 


§ 










d 




O 




hi 




d 




tn 




o 




1-1 




1-1 




•H 




JJ 




a 












> 




o 








u 


<>• 


dJ 


nJ 


u 


tn 


tn 


XJ 


nJ 




d 


d 




tn 


o 


0) 


d 




o 


6 


iH 


4J 



O 



•H I 
dJ B 
jZ o 



dJ 
tn 
o 
<x 
B 



dJ 
dJ 'a 
:3 

_ u 
1-1 



o 



M-l r-l 



^ > 



O O c^.l 
tnl 

d tn 



tn jc: dJ nl 

d dJ S 
o ^ 

a to ^1 

tn d 1-1 n 

OJ g M o 



Q » 



EKLC 



-78- 

104 




ERIC 



-79- 




ERIC 



106 

-80- 



X 



H 

I 



M 

H 
CO 

8- 
i 




ERIC 



-81- 

107 




ERIC 



-82- 

108 



H 



M 
H 
CO 

o 
u 

u 

o 



o 

s 

w 

Q 
Q 
M 



1^ 



SCHNECK 



YULDUZIAN 



WALLS 




oc 



0 tn 



c 

ID 
U 

>^ u 



3 
5 ^ 



PQ 











u 










o 


:j 






o 






iJ 












tn 




.H 


0) 








tn 








4J 




•r) 


•H 






U 




tn 


JJ 








tn 


a 








OJ 


> 




> 


.H 






(U 




iJ 


tn 


tn 


•H 


6 




tn 


dJ 


o 


o 




P 


a* -u 


o 







I 

o a 

tn o 

OJ iJ iJ 

tn (xj 

U C OJ 

c o 



iJ u 

6) O 

o a 

a o 

(U u 



I 

a iJ 

OJ 

tn B 

OJ 

u o 
o 



a 
<\) 
Td 

ij i-i (U 



OJ 

> 

OJ 

Td 



3 iJ 

U 

OJ 

Td o 



OJ 

Td 



tn ^ 



t3 ri 

tn o 

0) -u 

O U 

c o 

M > 



nd iH tn 

OJ iH I 

tn CTJ » 

OJ OJ ^ 

i-i c tn 

U OJ 

a OOH 

•H nj 

tn o • 

C iH (U 

(U (U B 

Td > 3 

3 OJ C 

tn 

OJ -H ^ 

> OJ OJ 

^ tn 



EKLC 



-83- 

109 



O 
H 

o 

M 

H 

to 



O 

o 
o 




ERIC 



110 

-84- 



SCHNECK 



YULDUZIAN 



WALLS 



o 

M 



a 

(U 
B 



OJ 

u 

H 



w 
c 
o 

•H 
JJ 

tn 

0) 
OJ 

6 
o 
o 
ij 

O 



f 



I 

I ^ 



QJ H 

CD M C 
(U (U o 

M C -H 

O ,0) -LJ 

a 00 nj 

•H U 

tn o c 

c iH e 

0) 0) S 
n3 > 

iJ .H 

tn 

u 

> 0) OJ v^ 

32 ^ d tQ 



o 
o 

3 iH 



I 

o 
u 

c 
o 



tn (tJ 

OJ OJ 

u a 

U OJ 

a 60 

iJ c^. 

tn o B «H 
C .H 
c: .H o -H 

n3 > c 

3 (U •H 



tn 



tn iH 

> (U OJ 



5 g 



60 
C 

tn 
tn 

OJ 

u 









.H 




iH 60 


tn 








re 


OJ ^ 
C t9 


u 


OJ e 


c 


60 1 


•H 


a 




^ O 


tn 


O -H 


ij 




a 


•H -H 


OJ 


OJ U 




> OJ 







tn .H tn 

n 3 .H 
0) -H 

> 0) OJ -H 

«^ jc w ^ 

D2 i-' 3 tn 



















OJ 








tn 
















OJ 


OJ 


ctJ 




u 


c 


C 




o 




O 




a 


60 


tn 




•H 




er 




tn 


O 






ij 




M 




a 


.H 


(U 




(U 




iJ 




Td 










OJ 


•S 




ij 


.H 






tn 






tn 




U 




.H 


OJ 


•H 




.H 


> 


OJ 


OJ 








tn 


^ 


32 


ij 


3 


tn 



X 



u 

o 

o 

tn 
tn 

OJ 

a 

OJ . 
tn 

cn 

n d 
M > 



0) » 
OJ iJ 

OJ O ( 

u I 
u c 
c o 



3 



tn 



60 cd 
o > 
u 



O c^. 



w tn 

n tn OJ 

5 Ji 2 

32 ^ .C, > 



-85-- 
111 




ERIC 



-86- 

112 



o 



CO 

g 



o 

H 

8 



SCHNECK 



YULDUZIAN 



WALLS 



o 

M 

O 

u 
C 

«J 

OJ 
M 
H 



o 



tn 

(U 

o* 

(U 

B 
o 
o 

o 



1 00 
. I 



loo 
I 



I CO 

I 



•H 

.OJ 
60 

C U 



! pa 

o 
u 

dJ 
M 
•H 

tn 

I a 

0) 
CO 
(TJ 
(1) 
M 
O 

c 



o 



c 

tn 

o 
a 



M 

O 



H 



a. a 



tn 
tn 
dJ 

dJ 
u 



o 

AJ 

D 
O 

60 



60 
C I 

tn U 
60 d 

tn 

60 'H 
T3 U 



0) 

o 
c 



I 

•H 
T3 

O 

tn 

OJ 
•H 

u 

•H 
1-4 
•H 



T3 

T3 
•H 
CTJ 

a 

O 

tn 

a M 
>s o 
ij > 

AJ T3 

u a 
0) c: 



I 

tn 



M ,H <J 



T3 


r 


I 




AJ 






OJ 




•H 




d 






tn 


•H 


T3 


tn 








T3 


d 






d 




dJ 




O 




dJ 






M 


M-l 


O 








O 


O 




AJ 




T3 




d 






tn 


•H 








OJ 


U 




T3 








60 


o 


(U 








tn 






M-l 


J3 


T3 




AJ 


OJ 




•H 


AJ 


•H 




c 






•H 








0) 










a 






o 




13 












tn 


C 


13 










0) 




d) 


o 




tn 




u 




i-i 








u 


d 


tn 




tn 








dJ 


d 




dJ 










o 


O 


a 






s 


dJ 


•H 


O 


>^ 


O 








AJ 


tn 


AJ 





















>^ 




T3 




N3 


U 








dJ 


AJ 








tn 


d 




a 






dJ 




d 




1 dJ 






d 














0 


0 




T3 




d 










•H 


dJ 


0 


% 






60M-I 






tn 


T3 








AJ 


dJ 


tn 


•H 




d 


•H 


AJ 






dJ 


:s 


d 


a 




T3 


0 


dJ 














AJ 




dJ 


0 




tn 
















tn 


<>• 


dJ 


•H 




dJ 




> 


dJ 




a 


u 




J3 


dJ 




0 


S2 


AJ 




AJ 





u 



ERIC 



-87- 

113 




-88- 

114 



r 



h-3 

8 

U 
t/i 

Q 
Q 



O 
H 

h-3 



SCHNECK 



.ULDUZIAN 



WALLS 



o 
u 



El 
H 



en 
C 
o 

•H 
JJ 

cn 

0) 

o* 

QJ 
B 
o 
u 
iJ 

O 



oc 



00 



00 



h 


1 






o 


c 






c 








ch 






cn .IS 


OJ 








iJ 






















u »o 




c 




d 




•H 












60 iH 


0) 








60 




iJ OJ 








C .H 


U 






tude 
know 


SO 


-s 


or 




.H 




tn 


o 


ctJ 




u 




U 


•H 




iJ 


•H 




> dJ 


u 


60 


cx 




03 O 
CX .H 


un 



I 

tn 
dJ 
u 



u I 
(U u 

ctJ tn 

H 



OJ 



"S ij Ij 



j:: tn 



O iJ 



tn 



dJ tn d 

60 t-i cd 
o 

iJ (1) iJ CO 

OJ ^ dj 

o u-( u 

-u i2 d 



I 

c 
o 

tn 

o 

0) 

p 

(U 
(U 

. 

a . 



tn 



trJ tn 

dJ tn 

u ci 
a 



u c 



I 

u 
d 

tn dJ 
d) d 



tn 

ij _ 
dJ H 
3 JQ 



^ w tn ^ 
^ (\) tJ) u u 
-HOGG 
O 4J CX 5 



Td dJ 

dJ iJ 

cn ffl 

dJ dJ 

t-i u 
a 

c o 



tn 

u 

H 

dJ 
S 4J 



tn 



ij ^ C 

J-i G 

G G -H 
CX 5 



(TJ g 



dJ C/) 

13 



C 'a 

dJ -H 

tn (XJ 

dJ cx 



tn >^ 



i-i i-i ij ' ^ 

60 > d) d) 

fJ g jC J3 . ^ 

^ X < 



tn -H 
* dJ ctf 
cx 



dJ iH 

ij ^3 

tn 

u 



cn CL. 

4-1 d) 

tn u-i tn 

dJ G iJ Q) 

dJ tn nd >^ 

dJ d iJ 

C (X cQ cn 

L» d) 

o G -H 



tn Td Td 

d fd u 

H cx 

i-i > 3 O 

^ -H G 

> d) d) nd nd tn 

:3.5-S g g 18 



-89- 

115 

o 

ERIC 



TABO 



STREIN 



STATLER 



SPINGLER 



REISTER 



o 



PFUHL 



CO 



HICKEY 



EVANS 



1-3 
O 

o 



CHELUCCI 



CHAPMAN 



BOWEN 



o ) 
a) 



a 



a 



oc 



oc 



oc 



BRADLEY 



a 

O 

u 

o 



H 



cn 
d 
o 

•H 
iJ 

w 

cy 
dj 

B 
o 
u 
ij 

o 



u I 

H O O 

iJ ^ 0) 

w u 

OJ >^ d 

cn u-i o 

«t3 tH cn 

QJ U 

M d (U 

u (U ^3 

H iH 

cn ^ -Lj 

iJ 03 3 

d ^ ^ 

(U o u 



cn 



T3 

> 4-» 
CTJ ^ 03 
123 CTj U 



M 

0) 

U cn 

(T3 g 

U dj 
iH 

(U ^ 

> O 

i-f M 

O <X 

cn 

o d 

iJ .H 

d 'r^ 
O 0 
•H I 

o 

a ^ 
M cn 

O -H 

U-I u 

d (U 



f 

O I 

<X OJ 

u 

T3 



u 

OJ ^ OJ 

ij o d 

iJ O CT] 



a* 



03 (U j2 



d 

rl _ 

O B U 

d C d 

cn OJ u-i 

o o 

iJ n_i 



c 

OJ 

cn iJ 03 

OJ i-H <U 

> -H C 

^ QJ 

w t3 4J 



d 

cn 
o 
o 



T3 

OJ cn 

cn Oi cn 

cd QJ M 

QJ iJ o QJ 



OJ 

u u 

OJ 03 03 



tT3 

go 
cn 

u u 

QJ »r4 

QJ 

^ J2 
4J 4J 

d 

(D tM 
S O 



•H U-« CJ 



QJ C 

U-I -H 
O iJ 

OJ T3 QJ 



d 

OJ 

d W) d M 

4J 'X3 TO QJ 

cn QJ 'X^ 

•H d 

QJ 5 (U cn h 

o ^ d 



H M 

QJ QJ 

42 OJ 

■ u 

OJ 

cn d 

OJ 



CTJ d o 
w i2 *j u 



d 

QJ C-. 

0 W) 

QJ C 

> -H 

O 

> 0 

d I 
d 
o 

►rl 

cn 

H 

u 

03 O QJ 



4J 

cn 0) 
> 

OJ 

> 



-90- 



X 



O 
O 

s 

CO 

§ 



SCHNECK 



YULDUZIAN 



WALLS 




u 


1 


u^ 






•H 


O 


o 








r-r 


u 






/■ — \ 


w 






u 








<>• 






u 


t-l 








M 








>> 




o 


0) 


cn 


U-l 


O 








•H 


W 








U 




> 


o 




a 


0) 






u 


dJ 


N 


o 


a 






•H 


w 




•H 


rH 




60 






•H 


o 




tn 


/ — \ 


4J 




•rl 


AJ 














/ — \ 






dJ 


o 


o 


o 


a 
1 




AJ 




•H 








AJ 


a 


u 








o 


cn 


AJ 


! 


g 






•H 






cn 




iH 




o 


•H 


> 




ij 


u-l 


o 








c 


0) 


DC 




u 




73 



X 



•H 


1 








0) 


o 


1 






-C 


a 




o 




AJ 




o 






ed 




> 


in 




cn 












dj 










d 




AJ 




u 


•H 


dJ 






o 


g 




o 


60 


a 














4J 


OO -H 




AJ 






cn 


cn 






H 


o 


AJ 








o 


a 










0) 


o 


o 


O 


-s 


T3 


4J 




cn 










u 




AJ 




dJ 


•H 


cn 


AJ 






dJ 




•H 










.H 


4-) 


4-1 


4J 


> 




a 


a 




Ha 


•3 


d) 

4-1 


me 


of 



X 



u 
















dJ 








u 




dJ 




dJ 




dJ 


dJ 














dJ 




AJ 


o 




c 


AJ 








AJ 


o 














4J 






o 




dJ 


cn 




a 


dJ 






19 


cx 


cn 




cn 








0) 


i-i 


u 




•H 




dJ 


4J 


o 


dJ 


dJ 






U 


cn 


4J 


dJ 


t-i 


AJ 




u 




o 




o 






c 


dJ 


cd 




a 


dJ 


<>• 








o 


•H 


a 


60 












dj 


a 


cn 




dJ 


d 


0) 


> 


•H 


AJ 


u-l 




•H 


AJ 


H 




a 


o 


AJ 






o 


cd 


dJ 






Td 


dJ 


> 


a 
1 




OJ 


Td 


dJ 


Td 


c 




60 




u 




•H 


c 


AJ 






dJ 


A^ 




o 


CO 


dJ 






cn 


d) 


•H 






a 






> 


cn 


dJ 




d) 


cn 60 


dJ 


•H 


•rl 


> 


o 




c: a 


> 


AJ 


O 








O •H 




O 


0) 






u 






cd 


Td 

















-91- 

117 

o 

ERIC 



T 



TABO 



STREIN 



STATLER 



SPINGLER 



►-3 
O 
O 

s 

to 
w 

Q 
P 
M 



REISTER 



PFUHL 



HICKEY 



8 



EVANS 



CHELUCCI 



CHAPMAN 



BO WEN 



BRADLEY 



vX 



oc 



oc 




X 



cn 

o :j 
a o 
ij 

dJ 

> 3 
n) ij 

ij 

cn -u 
nl 
C 

dJ dJ 

'S > 

ij 

cn 
o o 



tn 
dJ M 



nJ o 
o o 



tn 

tx o 

dJ ^ 

U iJ 

C nl 

O tH 

dJ u) 

G c nJ 



2 ^ 

B o 
ij 

dJ 
dJ n3 
> d 
nl ij 

ij 

cn -Lj 
ij nj 
C 

dJ dJ 
> 

iJ 
•H 

W 
O 

a 



a 

o 

•H Td 



dJ nl 
cn 

C c 

o dJ 

u s 



o 

c 
cn 

•H 

a 
a 
o 
u 

u cn 
ni ^ 
cn 

C nJ 

•H 4J 











ed 




dJ 




•H 


1 


cn 








dJ 


a 


o 




o 






o 


a 




a 


o 
ij 


iJ 


cc 


a 

•H 






dJ 


or 


nJ 


1 

uh 




dJ 






Td 


tH 




> 


d 






dJ 






ij 






W 






•H 


ij 










iJ 


•H 








cn 


iJ 


•H 


o 


o 




ij 


nJ 


•H 






cn 


c 




,Q 


> 


4J 


^ 


dJ 


dJ 


•H 




c 


cn 




> 


cn 


•G 


dJ 








C 


dJ 


B 


iJ 


ij 


iJ 


o 








cn 


•H 


<x 




cn 






cn 


tn 


C 


•H 


U 


o 


o 


dJ 




iH 


o 


Q 


a 




8 


a 





Id 



-92- 

118 



8 

H ^ 
23 



SCHNECK 



YULDUZIAN 



WALLS 




d 



o 



cn 




<>• 


u 






-a 




el 






> 










u 


U 




c 


o 




0) 


(U 


o 




u^ 


:2 




u^ 










tn 










> 






o 




o 


(-1 






IX 






B 






M 






M 






> 







AJ 

I -H 

cn 

nJ u 
o 

o 

a 0) 



> 



AJ 



o 

AJ C^. 

w q c 

H «9 AJ 

< AJ nj 



PQ 



cn 



> d 



cn AJ 
AJ nj 



3 

AJ 

cn 

o 
Q 



AJ 

•H ^ 
.H J-i 

nJ o 
cr 
u^ 

o o 

AJ 

cn 

AJ c 

<x o 

O AJ 

o .H 



cn 
o 

AJ 



nJ AJ o 



I 

C 
o 
u 

•H 
> 

c 
dJ 

c 



AJ 



C 

dJ dJ 
n3 > 



d) 

AJ 



o 

dJ n 

cn AJ 

o dJ 

o a 



u 
o 

c 

•H 

cn 

a 
e 
o 

o cn 
nj ^ 
cn 

C nJ 
•H oj 











XI 






-3 






d) 




dJ 




•H 


1 


cn 




u 




dJ 


a 


o 




o 






o 


a 




a 


to 


AJ 


cc 


a 

•H 






dJ 


or 


nJ 


1 

u^ 




dJ 




u^ 








> 








dJ 






AJ 


>^ 




cn 






•H 


AJ 










AJ 


•H 




u^ 




cn 


AJ 




o 


o 


<>• 


AJ 


(d 




•H 




cn 


c 






> 


AJ 




dJ 


dJ 






c 


cn 




> 


cn 




dJ 








c 


dJ 


a 


AJ 


AJ 


AJ 


o 








cn 


•H 


<x 




cn 






cn 


cn 




•H 


u 


o 


o 


dJ 






o 


Q 


a 






a* 





-93- 

119 

o 

ERIC 



TABO 



STREIN 



STATLER 



SPINGLER 



oc 



X 



REISTER 



PFUHL 



HICKEY 



EVANS 



CHELUCCI 



CHAPMAN 



BOWEN 



BRADLEY 



X 



X 





d 


d 














AJ 


3 






d 








Td 










o 












1 






O 












dJ 


dJ 




AJ 




u 


o 










dJ 


•H 










OJ 








> 


W) 


Td 


•H 




nJ 












tn 






AJ 






tn 










nJ 


d 


:j 




u 


u 










nJ 












nJ 


dJ 


d 






W) 


nJ 






u 


o 


o 








dJ 




dJ 


tn 


d 




dJ 




nJ 






d 




dj 




cn 








d 




t-i 




AJ 


dJ 


O 


O 










tn 


dJ 


d 






a 








•H 




o 




nJ 


o 


•H 




O 








AJ 




o 


to 




o 


0) 








tn 


d 


o 


O 


t»i 


AJ 




d 


a* 






d 


dJ 


•H 


a 




B 


U 


a 


<>• 


dJ 


iH 




AJ 


o 










dJ 


o c- 




dJ 


u 


AJ 


0 




dJ 


•H 






dJ 


iH 






•H 


O 


B 


•H 




(-1 


u^ ^ 




nd 




nl 








tn 


g 




CO 




tn 


>^ 




tn 






tn 












o 


8 






dJ 




o 






AJ 


AJ 


/-^ 




o 


a 


4J 


tn 


bO O 








d 






w 










CO 


d 


•H 




dJ 






d 


iH 


•S 




tn 


o 


Td 


to 




u 




dJ 




u 




dJ 


•H 


^— ' 


N 


d 


§ 


d) 


iH 










d) 






a 




i-f 




o 


too 








•H 










>H 60 




>^^d 






tn 


dJ 


dJ 








d 








iH 










d 






d) 


u 


iJ 


u 




tn 


tn 






•H 


AJ 






d 




AJ 


tn 




AJ 




o 


dJ 




nl 




nJ 








AJ 


tn 




u^ 


iJ 






tn 




a. AJ 


d 


s 


ni 




tn 


dJ 




dJ 


o 


O 


dJ 


AJ 




u 












u 


ni a 


dJ 




•H 


AJ 


dJ 


>^ 


tn 


u 




O 


tn 


dJ 


dJ 




AJ 




4J 




dJ 




dJ 


§ 




a 


M 


o 






a 


a 


-C 


nJ 




> 


dJ 


d 




d 


•H 


> 


dJ 


^ o 








d 


nd 


U 


o 


a 




o 


dJ 








dJ 


O 


o 








,01 o 


o 


33 


d 






O 


o 






tn 






33 


AJ 






a 


a 


33 








dJ 




:5 


•H 










u 
























< 


dJ 


d 


o 


t-i 










a 


o 




















Pl4 








<X 










M 
















PQ 

















ERIC 



-94- 

120 



M 

H 



H 



o 

M 
H 
CO 
U3 
t3 

o 
o 




121 



-95- 



ERIC 




122 

-96- 



ERIC 



T 




ERIC 



123 

-97- 



X 

a 

H 



H 

a 

s 




ERIC 



-98- 

121 



M 



o 

H 



O 
M 

H 

CO 



U 

o 



h-3 
O 
O 

g 

O 



o 

H 

•-3 



— 


1 


















1 
































] 







































SCHNECK ' 

1 V 


x> 1 








YULDUZIAN ° 


1 
1 

» 1 








a 

WALLS j 


j 
1 









O 

u 

C5 



M 
H 



en 
o 

•rt 
iJ 

w 
o* 

0) 

B 
o 
o 

iJ 

o 



u^ 

U 

a 



o cn 
u > 



u u 

tn 

M 

PC t3 



^ -C 
M u 
O 

U 

cu o 
tn 

a 

o o 

ij 

tn tT3 



u^ 
o 

Ui 



U 

O 

a 



125 
-99- 



ERIC 



APPENDIX B 



ETS SECOND QUARTERLY REPORT 



126 



Quarterly Report 

"Career Education Instructional System Project" 

Newark School District 
Newark , Delaware 

USOE Grant No. OEG'-74-0955 

Project No. VOOOIVW 

The information in the quarterly report is based upon Educational 
Testing Service's involvement with the Newark, Delaware Career Education 
Instructional System Project between April 1, and June 30, 1975. ETS's 
evaluation activities during this period were primarily concerned with 
the collection and preliminary analysis of CEIS project related data for 
formative evaluation purposes and as input for the development of an eval- 
uation design. Also, the data collected were used to develop the Outcome 
Question / Treatment tables. 

A fifteen item questionnaire was developed by ETS for self-adminis- 
trating to all teachers involved with the CEIS project. On May 15 and 16, 1975 
ETS evaluation staff visited the CEIS project direction staff and each of 
the school'CEIS coordinators to discuss the data collection activities- 
ETS staff explained to the coorainators how to complete the questionnaire 
and additional copies were provided for each of the CEIS teachers within 
the schools. The coordinators were requested to complete the question- 
naire by May 23, 1975 and to forward the completed questionnaires to the 
CEIS project direction staff for return mailing to ETS. 

Due to close of school activities the return date for the question- 
naires was extended to June 20, 1975. As of this date 91% (N=51 teachers) 
of the 57 teachers involved in the CEIS project returned their question- 
naires. A discussion of the questionnaires* findings and preliminary 
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analyses follows. 

In order- to learn what sources of information have been provided 
by the CEIS project staff related to implementation of the CEIS curri- 
culum units the teachers were asked to identify specific sources of in- 
formation. Table 1 shows the number and percent of teachers using spe- 
cific sources of information. 

Table 1 

Number and Percent of Teachers Using Specific Sources of Information 

Number Percent of 

of Teachers 

Teachers Responding 

1. Copies of reports from career 

education projects re imple- 37 -j^y 
mentation 

2. Textbooks , periodicals , journals 36 72%- 

3. Relevant reference materials 34 68% 

4. Periodic meetings and discussions 

with project staff 33 66% 

5. Project memoranda 

6. Instructional manuals 

7. School level administrative staff 



32 64% 
30 60% 



20 40% 
8. District director of curriculum or 

instruction 18 36% 

9.5 District level administrative staff 17 
9.5 Media presentations 27 

11. District-wide meetings with instruc- 
tional staff involved in the project 

12. Test materials and manuals 

13.5 University Staff ^ 
13.5 Others: 



34% 
34% 

32% 

12 24% 
8% 



4 



Total N of teachers =» 50 
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Not unrelated to sources of information provided by the CEIS project 
was the feedback requested from the project staff concerning the develop- 
ment and implementation of the curriculum units by the teachers. As shown 
in Table 2 the distribution of responses concerning the amount of feedback 
requested approximates a normal distribution. Six percent (N = 3 teachers) 
of the teachers indicated that an extensive amount of feedback was requested, 
40% (N = 40 teachers) a moderate amount, 48% (N* = 48 teachers) a minimal 
amount and 6% (N = 3 teachers) indicated that no feedback was requested. 

Table 2 
CEIS Project Feedback 

Amount of Feedback Number Percent of 

of Teachers 
Teachers Responding 

Extensive amount of feedback 3 5% 

Moderate amount of feedback 20 40% 

Minimal amount of feedback 24 48% 

No feedback requested 3 
Total N of teachers = 50 

The teachers were also asked to indicate the number of times during the 
school year 1974-75 that they were contacted by the CEIS project staff. The 
median response reported by the teachers was 3 contacts. This suggests that 
the teachers were contacted by the project staff about once every 15 weeks 
during the school year 1974-75. The teachers also reported that the CEIS 
project staff conducted staff meetings about twice a year. 

In order to provide information to the CEIS project staff about the over- 
all quality of the 195 curriculum units developed and implemented the teachers 
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were asked to indicate the relative degree of modification needed to im- 
prove the quality of the units. The areas of modification were: (1) format, 
(2) style, (3) goals and objectives, (4) instructional methodology, (5) con- 
tent, and (6) evaluation. Each of the teachers indicated the relative degree 
of modification required ranging from extensive modification to no modifica- 
tion. Table 3 indicates that 91% of the teachers reported that the curri- 
culum units format required no modification, 90% reported that the units style 
required no modification, 86% reported that the units goals and objectives re- 
quired no modification, 85% reported that the units instructional methodology 
and content required no modification and 77% reported that the units evalua- 
tion procedures required no modification. 

Table 3 

Degree of Curriculum Uni ts Mo dification Requir ed^ 

Degree Required ^ 

Extensive 

Areas of Modification No Modification Modification 





Number 
of 

Teachers 


Percent of 

Teachers 
Responding 


Number 
of 

Teachers 


Percent of 

Teachers 
Responding 


Format 


43 


91% 


4 


9% 


Style 


42 


90% 


5 


10% 


Goals and Objectives 


42 


86% 


6 


14% 


Instructional Methodology 


31 


85% 


7 


15% 


Content 


41 


85% 


7 


15% 


Evaluation 


36 


77% 


11 


23% 



Total N of teachers = 50 
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The teachers were also asked to indicate their opinion of how well their 
students liked the curriculum units. As displayed in Table 4, 70% of the 
teachers reported that the students liked the units very much and 30% repor- 
ted that the studen::s liked the units somewhat. None of the teachers repor- 
ted that the students were displeased or turned off by the units. A compari- 
son of the percentages between the elementary and middle schools is interesting. 
Eighty-five percent (N = 22 teachers) of the elementary school teachers repor- 
ted that their students liked the units very much as compared to 52% (N = 13 
teachers) of the middle school teachers reporting in the same category. Like- 
wise, only 15% of the elementary schoql teachers reported that their students 
liked the units somewhat as compared to 44% of the middle school teachers. 

Table 4 

Teachers"Opinions of How Well S tudent s Liked Curriculum Units 

^Elementary School Middle School 
Teachers Teachers 

Totals 

Number Percent of Number Percent of Number Percent of 
of Teachers of Teachers of Teachers 
Teachers Responding Teachers Responding Teachers Responding 

R es pons e Cat eg or y 

Students liked units 22 85% 13 55% 35 70% 

very much 

Students liked units 4 15% 11 45% 15 30% 

somewhat 

Students were gener- 0% 0% 0% 

ally displeased but 
tolerated units 

Students were turned 0% 0% 0% 

off by units 

Total N of teachers = 50 

When the teachers were asked the question: Was any of the material covered 
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by the units presented to the students prior to the implementation of the 
CEIS project? 62% (N = 31 teachers) responded by saying yes. Table 5 on 
the following page illustrates the teachers estimate of the percentage of 
materials covered for each of the schools participating in the CEIS project. 

The mean percent of the materials covered for the teachers responding yes, 
the mean percent across all teachers, and the range of percents for elemen- 
tary and middle schools are presented below. 

i 

Elementary Schools Middle Schools Total 

X Percent 

39% 33% 

34% 26% 
0%-100% 0%-100% 

The data presented in Table 5 and above indicates that approximately 
2 6% of the material covered in the units developed by the CEIS project was 
presented before the project was implemented. At the middle school level 
34% of the material was covered before the project was implemented as com- 
pared to 21% at the elementary school level. Ogletown Middle School re- 
ported the highest percent covered (percent = 42%) and Cobbs Elementary 
School the lowest percent (percent = 9%). 

The responses to the question: What specific types of assistance would 
you like to see the project direction staff make available to you for the 
development and implementation of the career education curriculum units? are 
found in Table 6 on page 8 • The types of assistance requested are rank 
ordered from the most frequently to the least frequently requested. As shown 
in Table 6 the most frequently requested type of assistance was funds for 



Responding 

Yes " 30% 

X Percent 
All 

Teachers 21% 
% Range 0%-75% 
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purchasing special curriculum materials, equipment or supplies. The least 
frequently requested type of assistance was greater professional supervi- 
sion of teachers implementing the curriculum units. 

Table 6 

Project Assistance Re que sted By Tea chers 

Number of 



Percent of 



Type of Assistance 

1. Funds for purchasing special curriculum 
materials, equipment or supplies. 

2. Release time for developing and refining 
current curriculum units. 

3. Inservice preparation for instructional 
staff. 

4. Resource personnel in career education. 

5. Greater professional supervision of in- 
structional staff. 



Teachers 
38 

26 

25 
20 

12 
4 



Teachers 
Responding 

76% 

50% 

50% 
40% 

24% 
8% 



6. Others: Teacher Aides. 
Total N of teachers = 50 

The teachers were also asked to identify constraints that might pre- 
vent them from implementing the units in their classroom. Table 7 reveals 
that 60% (N = 30 teachers) of the teachers reported that funds are needed 
to purchase special supplies and equipment before the units can be implemen- 
ted. The second most frequently reported constraint (N = 16 teachers) was 
the need for inservice teacher training prior to unit implementation. Only 
one of the teachers reported that lack of direction and guidance by the 
CEIS project staff prevented the units from being implemented. 
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Table 7 



Constraints Preventing Implemen tati on of Curri c ulum Unit s 

Number Percent of 

of Teachers 
Types of Constraints Teachers Responding 

Funds needed to purchase special 

supplies and equipment before the 30 60% 

units can be implemented. 

Specialized teacher preparation 

required before the units can be 16 32% 

introduced. 



The implementation of the units 
takes too much time away from 
other subjects. 



5. The school administration is not 
supportive of the implementation 
of the units. 

6. Lack of direction and guidsmce by 
the project staff. 

7. Directions are not clear as to how 
the units are to be implemented. 

Total N of 50 teachers = 50. 



16% 



4. The units were poorly developed 
and are inappropriate for use 

in the classroom. ^ 



6% 



2% 



0% 



On the following page. Table 8 displays the instructional methods 
used by CEIS teachers in presenting the curriculum units. The number and 
percent of teachers using each of the types of methods in elementary and 
middle schools and the total number and percent across both school levels 
are presented. 
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Table 8 

Instructional Method s Used by CEIS Teachers 



Elementary Schoo l 
Level 



_Mlddle School 
Level 



Totals 





Types of Methods 


Numb e r 
of 

Teachers 


Percent of Number Percent of Number 

Teachers of Teachers of 
Responding Teachers Responding Teachers 


Percent oJ 
Teachers 
Responding 


1. 


Class Discussions 


26 


100% 


22 


92% 


48 


96% 


2. 


Audio Visual Materials 


25 


96% 


18 


75% 


43 


86% 


3.3 


Teacher Demonstrations 


16 


62% 


21 


88% 


37 


74% 


3.5 


Self-Directed Study 


21 


81% 


16 


67% 


37 


74% 


5. 


Teacher Lectures 


19 


73% 


17 


71% 


36 


72% 


6. 


Student Work Sheets 


14 


54% 


19 


79% 


33 


66% 


7. 


Bulletin Board 
Displays 


21 


81% 


9 


38% 


30 


60% 


8. 


Visitors 


20 


77% 


7 


29% 


27 


54% 


9, 


Role Playing 


16 


62% 


10 


42% 


26 


52% 


10.5 


Field Trips 


16 


62% 


9 


38% 


25 


50% 


10.5 


Curriculum Infusion 


16 


62% 


9 


38% 


25 


50% 


12. 


Group Counseling 


7 


27% 


9 


38% 


16 


36% 


13. 


Simulation 


7 


27% 


7 


29% 


14 


28% 


14. 


Testing 


3 


12% 


6 


25% 


9 


18% 


15. 


Work Study 


0 


0% 


4 


17% 


4 


8% 


16. 


Career Days 


1 


3.9% 


2 


8% 


3 


6% 


17. 


Tn t* ^TTi 'i no 


u 


0% 


2 


8% 


2 


4% 


18.5 


Career Clubs 


1 


3.9% 


0 


0%. 


1 


2% 


18.5 


Co-operative Education 


0 


0% 


1 


4% 


1 


2% 


20. 


Others : 
Assemblies 
Art Projects 
Models 
Reading Rack 


2 


7.7% 

136 


1 


4% 


3 


6% 
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For both school levels the five most frequently reported methods of in- 
struction are class discussions, audio visual materials, teacher demonstrations, 
self-directed study and teacher lectures. Analysis between the levels reveals 
some interesting comparisons. For example, 68% of the elementary school teachers 
reported using field trips and curriculum infusion methods of instruction as com- 
pared to only 38% of middle school teachers. 

In order to determine the relative degree of emphasis of the CEIS project 
the teachers were asked to indicate which of the nine student outcome statements 
(goal areas) listed in the Federal Guidelines for Evaluation were covered or re- ' 
lated to each one of the curriculum units developed. Table 9 on the following 
page displays the frequency of responses in each of the nine student outcome 
statements (goal areas) as judged by the teachers. 

As shown in Table 9, 84% (N = 130 units) of the 155 units developed re- 
late to improving basic academic and vocational skills with approximately the 
same percentage found at the elementary and middle school levels. Awareness and 
knowledge of the world of work is covered by 79% (N = 120 units) of the units. 
Seventy-one percent of the units at the elementary school level deal with aware- 
ness and knowledge of the world of work and 86% at the middle school level. 
Seventy-eight percent (N = 120 units) of the units deal with awareness of work, 

values and the desire to work. At the elementary school level 73% of the units 
deal with this student outcome statement as compared to 83% at the middle school 
level. Competency in career decision niaking is covered by 77% (N = 118 units) 
of the units with 72% of the elementary units and 81% of the middle school 
units covering this area. Increased self awareness is found in 48%(N = 71 units) 
of the units. Forty percent of the elementary school units .and 55% of the 
middle school units relate to this outcome area. Less than 21% (N = 35 units) 
relate to work seeking and work getting skills, placement in an occupation or 
further education, or awareness of means of continued education. The student 



outcome statements noted with an asterisk require third-party evaluation 
as specified in the Federal Evaluation Guidelines. 

Table 9 

Student Outcome Statem ents (Goal Ar eas )_RpT^£pHrn Curriculum Units 



Student Outcome Statements 



_Elementary S choo_l . Middle School 
Level Level 



Totals 



Number Percent of Number Percent of Number Percent 
of Units of Units of Units 

Units 



Units 



of 
Units 



* I. Increased Self 

Awareness 

II. Basic Academic and 
Vocational Skills 

III. Awareness of Work 

Values-Desire to work 

* IV. Awareness and Knowledge 

of World of Work 

* V. Competency in Career 

Decision Making 

VI. Good Work Habits 

VII. Work Seeking and 

Work Getting Skills 

VIII. Placed in Occupation 
or Further Education 

IX. Awareness of Means of 
Continued Education 



Total number of units at elementary level 
Total number of units at middle level 



36 

76 

67 

65 

66 

27 
6 



40% 

83% 

73% 

71% 

72% 

30% 
7% 

0% 



35 
54 
53 

55 
52 

8 

15 



55% 
84% 

83% 

86% 

81% 

12% 
23% 

0% 

3% 



71 

130 

120 

120 

118 

35 
21 



48% 

84% 

78% 

79% 

77% 

21% 
15% 

0% 

1.3/' 



91 
64 



The previous section of the report discusses ris's evaluation activities 
related to the collection and analysis of data for formative evaluation pur- 
poses. ETS has also used the data collected for the development of th - Out- 
come Question/Treatment tables. Since the teacher questionnaires were not 
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returned until shortly before the end of the second quarter, ETS has not 
as yet completed the tables and therefore they are not Included as work 
products for the second quarter. 

ETS is presently using the information collected through the teachers' 
questionnaires for developing an evaluation design. 

Conceptually, ETS's preliminary plans call for a tailor made evaluation 
design. Using this design each class involved in the project (presented a 
curriculum unit) will be assessed only on those student outcomes which the 
unit has been shown to be related. Also, to find out if the units have any 
effects on student behavior each class will be assessed immediately before 
and after the presentation of the units. This design is proposed in place -f 
of the more typical fall and spring pre and post testing design in order to 
increase the probability of assessing the effects of the units. 
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APPENDIX C 



MEASURES REVIEWED FOR USE 
IN CEIS PROJECT 
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Measures Reviewed for Possible Use 
in CEIS Project FY 75-76 

The instruments listed below were chosen for review based upon annotated 
bibliographies of the ETS test collection. Additional instruments were 
rejected without review as (1) inappropriate for the age groups to be included 
in the evaluation of the Career Education Instructional Systems Project, or 
(2) not measuring the project objectives. 



Self-Awareness - Grades 3,6,8 

"Self-Esteem Inventory", Stanley Coopersmith. In The Antecedents of Self- 
Esteem by Coopersmith; W.H. Freeman and Co., San Francisco, 1967. 

Comment: No grade level norms; insufficient technical data. 

Self ^Report - Inferred Self -concept Scale" ("About Me"). James Parker, 
in " The Relationship of Self Report to Inferred Self-concept" by 
James Parker in Educational and Psychological Measurement, 1966, 26, 
291-700. 

Comment: No norms data, no reliability reported. 

How I See Myself," Ira J. Gordon, Florida Educational Research and 
Development Council, University of Florida, Gainesville, 1968. 

Comment: Well documented, but response made is more difficult 
for young children than that used in the Piers-Harris 
scale. 

" Piers-Harris Children's Self-concept Scale "("The Way I Feel About Myself"). 
C. Piers and D. Harris, Counselor Recordings and Tests, Nashville, 
Tennessee, 1969. 

Comment: Well documented, reviewed favorably in Euro's Mental 

Measurement Yearbook , 5th Edition. Format requires only 
a yes-no response. 

Career Knowledge - Grades 3, 6 

Career Education Cognitive Questionnaires", B. Rader and K. Nelson, 
Minnesota Research Coordinating Unit for Vocationa] Education, University 
of Minnesota, Minneapolis, 1975. 

Comment: Field testing involved a relatively limited sample, but 

instruments are presently being used to evlauate a number 
of Part D projects, and additional data should be available 
soon. Quality of printing on instruments is not uniformly 
high. Recommended by USOE Guidelines for Evaluation. 
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" Career Education Needs Assessmnnt, " A. Blome and G. Rask, Olympus 
Publishing Co., Salt Lake City, 1975. 



Comment: Good face validity in instrument for grades 4-6, but no 
technical data was available, K-3 instrument requires 
individual administration. 

Career Knowledge and Decision-Making - Grades 4-6 

"Readiness for Vocational Planning" by Donald Super. In Emerging Careers 
by Warren D. Gribbons and Paul R. Lohnes, Teacher's College Press. 
Columbia University, New York, 1968. 

Comment: Requires individual interviews, not appropriate for 
evaluation. 

"Career Development Inventory," Donald Super and David J. Forrest. Teacher's 
College, Columbia University, New York, 1972. 

Comment: Not published; requires weighted scoring; includes atti- 

tudinal and cognitive scales; data available from only one 
study of 400 tenth grade students in Michigan. 

"Guidance Inquiry," M.Katz. ETS, Princeton, N.J. 

Conmient: No longer available: redesigned as an instructional pro- 
gram. 

" Assessment of Career Development, Grades 9-12," American College Testing 
Program, Houghton Mifflin Company, Atlanta (Boston), 1974. 

Comment: Well documented, more comprehensive than others reviewed, 
good match of project objectives and test subscales; 
requires 125 minutes of test time. Recommended by USOE 
Guidelines for Evaluation. 



